Golang 速查表 Cheatsheets


原文链接: Golang 速查表 Cheatsheets

Golang

General

Binary file, that includes all dependencies.

GOPATH
   ├── bin/
   ├── pkg/
   │    └── linux_amd64/
   └── src/
       ├── github.com
       │      └── oleglegun/
       ├── golang.org
       └── gopkg.in
/*
Every package should have a package comment.
Package ... implements ...
*/
package main // executable main must contain function `main`

func main() {
    println("Hello")  // println works w/o any libs
}

// Do makes ...
func Do() {}

gb

Vendoring - save dependencies within repository.

vendor/src/my-lib folder can keep all local vendoring files.

alias ggb='$GOPATH/bin/gb'

Files structure with gb:

project
   ├── src/
   └── vendor/
       └── src/
            └── github.com/
CommandDescription
gb vendor fetch gopkg.in/telegram-bot-api.v4Download package to local vendor folder
gb vendor updateUpdate all local packages
gb vendor restoreInstall all packages listed in manifest
gb buildBuild project from src/

All info about packages is located in manifest

Go Tool


go run main.go # build + execute program
go build main.go; ./main # build + execute separately
go get -u github.com/user/repo

# cross compilation
env GOOS=linux GOARCH=amd64 go build -o ./bin/cmd ./src/

go doc
go doc ./dir
go doc ./dir.FuncName
godoc -http=:6060

Makefile Example

goimports formats code + manages insertion and removal of import
declarations as needed.

import

import "github.com/oleglegun/repo"  // path relative to GOPATH (accessible only in current file)
import _ from "github.com/..."      // import package for its side effects

func main() {
    repo.Method()  // Capitalized method == exported
}

In Go only Capitalized var/const/func are exported (accessible
throughout the whole package (often folder)).

Scope

  • syntactic block - sequence of statements enclosed in braces
  • lexical block - block that is not explicitly surrounded by braces

Scopes in Go:

  • universe block (entire source code, lexical block)
  • package level block
  • file level block
  • for, if, switch blocks
  • blocks for each case in switch/select
  • each explicit syntactic block

In Go, package is a unit of encapsulation, not a type.

Constants

Only numbers, strings/characters and booleans can be
constants. Expression that defines constant must be constant, because it
evaluates at compile time. Can be used to set a fixed array size.

const i int64 = 10
const name = "Oleg"

const (
    flag1 = true
    flag2 = false
)

const x := math.Sin(0) // Wrong, Sin() needs to happen at run time

iota - generator (auto-increment)

Excellent for enum.

iota resets in each block ().

const (
    one =  iota + 1 // 1
    two             // 2
    _               // skip 3
    four            // 4
)

const (
    _ = iota * 2    // 0
    i2              // 2
    i4              // 4
    _               // skip 6
    i16 = iota * 4  // 16 - redefine
    i32             // 32
)

Used for checking flags w/ mask.

// Masks
const (
    flagUserIsActive = 1 << iota
    flagUserIsLogged
    flagUserIsSubscribed
)

var userFlags = '\x11'

println("User is active", userFlags & flagUserIsActive)

Variables

Short Notation assignment for new variables

Use for working w/ functions: age := getAge() when we know the return
type and that there won't be any problems w/ dynamic type assignment.
:= can be used only within a function, not for package-level
variables.

age := 25

res1, err := do()
res2, err := do() // in := assignment `err` can be reused even if it has been already declared

Multiple Assignment

Multiple assignment should be used only when it helps readability
(initialization in for loop).

var i, j, k int = 1, 2, 3
var s1, s2 string = "Hello", "World"

// 2nd variant - often used for global (package) variables
package main

var (
    i int = 1
    j     = 2
    s1    = "Hello"
)

// Tuple assignment (swap values)
// All of the right-hand side expressions are evaluated before any of the variables are updated
i, j = j, i
a[i], a[j] = a[j], a[i]

Default values

All variables in Go are initialized by default w/ nil values. We can
define variable and instantly use it w/o value assignment. Go has
default values for all variables (unlike C, where variable w/o value
can point to memory addr w/ trash).

int: 0
float: 0
string: ""
bool: false
*pointer : nil

Bitwise operations

 var x uint8 = 1<<1 | 1<<5
 var y uint8 = 1<<1 | 1<<2

 fmt.Printf("%08b\n", x)    // "00100010", the set {1, 5}
 fmt.Printf("%08b\n", y)    // "00000110", the set {1, 2}
 fmt.Printf("%08b\n", x&y)  // "00000010", the intersection {1}
 fmt.Printf("%08b\n", x|y)  // "00100110", the union {1, 2, 5}
 fmt.Printf("%08b\n", x^y)  // "00100100", XOR the symmetric difference {2, 5}
 fmt.Printf("%08b\n", ^x)   // "11011101", each bit is inverted
 fmt.Printf("%08b\n", x&^y) // "00100000", AND NOT the difference {5}

 for i := uint(0); i < 8; i++ {
     if x&(1<<i) != 0 {     // membership test
         fmt.Println(i)     // "1", "5"
} }
 fmt.Printf("%08b\n", x<<1) // "01000100", the set {2, 6}
 fmt.Printf("%08b\n", x>>1) // "00010001", the set {0, 4}

^x - bitwise negation or complement (unary operation)

Types

  • Basic types: int, float, string, bool
  • Aggregate types: array, struct
  • Reference types (indirect refer): slice, map, pointer,
    function, channel
  • Interface types

Struct, array and interface may contain reference types.

Basic Data Types

int, float, bool

// INT
var i int = 10                // 32/64bit int, depends on platform
var autoInt = -10             // Auto assign type (int)
var bitInt int64 = 1<<32 - 1  // int8, int16, int32, int64
var uInt uint = 100           // unsigned 32/64bit int, depends on platform
var uBigInt uint64 = 1<<64 -1 // uint8, uint16, uint32, uint64

// FLOAT
var pi float64 = 3.14         // float32, float64 (precision)

// BOOL
var b = true                  // boolean

Use unsigned integer only when it is absolutely necessary
(hashing, cryptography, bit sets, parsing binary file format...

float64 is preferable for most purposes because float32 computations
accumulate error rapidly.

Short-circuit behavior

s != "" && s[0] == 'x' // if left is false => right will not be evaluated

string

All strings are in UTF-8.

String values are immutable.

Immutability means that it is safe for two copies of a string to share
the same underlying memory, making it cheap to copy strings of any
length. Similarly, a string s and a substring like s[7:] may
safely share the same data, so the substring operation is also cheap.
No new memory is allocated in either case.

var str1 string = "Hello\n\t" // double quotes for string literal
var str3 = `Hello
            there`            // Raw string literal (invisible symbols \t,\n)

// BINARY/'SYMBOL'
var binary byte = '\x28'      // single quotes for symbols

Raw string literals are useful for RegExp, which tend to have lots
of backslashes.

var str1 = "Hello"
var str2 = ", world!"
str := []string{str1, str2}

// Length
println(len(str1))                    // 5 - number of bytes
println(urf8.RuneCountInString(str1)) // 5 - number of runes


// Concatenation

// 1. `+` - Use only on small number of short strings: too many allocations
str3 := str1 + str2

// 2. strings.Join() - Use for large slices of strings
strings.Join(str, "")

// 3. bytes.Buffer - most efficient
var buffer bytes.Buffer
for _, str := range str {
    buffer.WriteString(str)
}
fmt.Println(buffer.String())


if str1 == str2 {} // Equality
if str1 > str2 {}  // Compare

Unicode

Unicode is embedded in Go.

1st Byte2nd Byte3rd Byte4th ByteFree BitsMaximum Unicode Value
0xxxxxxx7007F hex (127)
110xxxxx10xxxxxx(5+6)=1107FF hex (2047)
1110xxxx10xxxxxx10xxxxxx(4+6+6)=16FFFF hex (65535)
11110xxx10xxxxxx10xxxxxx10xxxxxx(3+6+6+6)=2110FFFF hex (1,114,111)

rune (Unicode code point) - is an alias for int32 and is equivalent
to int32 in all ways. It is used, by convention, to distinguish
character values from integer values.

var symbol rune = 's'
unicodeSymbol := ''
unicodeSymbolFromCodePoint := '\u2222'

A rune whose value < 256 may be written with a single hexadecimal escape
\x40, but for higher values, a \u or \U escape must be used.

Iterate on symbols

strRu := "Привет"
for index, rune := range strRu {  // range loop performs UTF-8 decoding implicitly
    fmt.Printf("%#U %d", rune, index)
    println()
}

/*
U+041F 'П' 0
U+0440 'р' 2
U+0438 'и' 4
U+0432 'в' 6
U+0435 'е' 8
U+0442 'т' 10
*/

For unexpected input byte DecodeRuneInString or range loop
generate special Unicode replacement character \uFFFD <?>.

Iterate on bytes

str := "Привет"
slice := []byte(str)  // convert String to Byte array

for index, value := range slice {
    fmt.Println(index, value)
}

Composite Types

Array

Is a low-level data type. Size is part of array type => can't change its
size. The types [10]int and [20]int are distinct.

Array is a value type. Assigning one array to another copies all the
elements. Passing array to a function - creates copy of it. Using a
pointer to an array is efficient and allows the called function to
mutate the caller's variable.

var arr1 [4]int // [0 0 0 0] create array + allocate memory for its elements

const size uint = 2
var arr2 [size]bool // [false false]

arr3 := [...]int{1, 2, 3} // [1 2 3], len = auto (3)

// Edit value of an element
arr3[2] = 10 // [1 2 10]

// Multidimensional Array
var arr4 [2][2]int // [[0 0] [0 0]], len = 2

Slice

Slice is an array w/ dynamic size. Slices hold references to an
underlying array, and if you assign one slice to another, both refer to
the same array.

Pointer points to the first element of the array that is reachable
through the slice, which is not necessarily the array's first element.

Passing a slice to a function permits the function to modify the
underlying array elements.

Slices unlike arrays are not comparable, we can compare only []byte
using bytes.Equal()

struct {
    length          // length ≤ capacity
    capacity        // doubles on each length overflow
    pointerToArray  // pointer to the first element of a underlying array (fix size)
}
// Create slice
var sl1 []int               // []
sl1 := []int{1, 2, 3}       // [1 2 3]
bsl := []byte{0x0, 0xFF}

// Memory efficient creation (we know the len/cap beforehand)
sl2 := make([]int, 5)       // [0 0 0 0 0] len = 5 cap = 5!!
sl3 := make([]int, 0, 10)   // [] len = 0 cap = 10

// Length
len(sl1)  // 3

// Capacity
cap(sl1)  // 4

// Slicing (shrink)
sl := []int{1, 2, 3, 4, 5}
sl[1:3]  // [2 3]
sl[1:]   // [2 3 4 5]
sl[:3]   // [1 2 3]

// Add element (grow)
sl1 = append(sl1, 100)
fmt.println(sl1)      // [1 2 3 100]

// Concat another slice
sl := append(sl1, sl1...) // ... - spread
fmt.Println(sl)           // [1 2 3 100 1 2 3 100]

// Copy slice
slCopy := make([]int, len(sl))  // new slice must have the same len
copy(slCopy, sl)

// Append slices
sl := append(sl1[:2], sl2[1:]...)

// Slice from Array
arr := [...]int{1, 2, 3}
sl := arr[:]     // sl: [1 2 3]
arr[1] = 10      // sl points to the same addr as arr
fmt.println(sl)  // [1 10 3]

Map

Map hold reference to an underlying data structure (created by make).
When a map is passed to a function, it receives a copy of the reference
=> any changes will be visible through the caller's map reference too.

Map does not guarantee key order.

Keys of map can be of any comparable type. Value can be of any type at
all.

We can't take the address of a map element because growing a map might
cause rehashing of existing elements into new storage locations,
invalidating the address.

var m map[string]string  // nil - points to nothing
//     |     |     |
//   type   key  value

// Nested Map
var mm map[string]map[string]string

// Initialize
var m map[string]string = map[string]string{}  // Empty Interface {}

// Initialize w/ `make`
var m = make(map[string]string)

// Short notation for Initialization
m := map[string]string{}

// Add element
m["firstName"] = "Oleg"  // String literal as key
m[varName] = "string"    // Variable as key

// Read element
name = m["firstName"]

// Read nonexistent element
name = m["lastName"]     // "" - returns default zero value for a type

// Check if key exists
_, ok := m["lastName"]         // only check for key existence
lastName, ok := m["lastName"]  // returns value + ok - bool

// Delete element
delete(m, "firstName")

, ok - "comma ok" idiom

Control Flow

if

Checks strictly for bool value. Doesn't support implicit type
conversion (1 ≠ true).

if flag {
    println("flag is true")
}

if a == 1 && flag {
    println("success")
}


// Complex if using `;`
// firstName, ok are only accessible in 'if' scope

if firstName, ok := m["firstName"]; ok {
//     |                             |
//  can be `_`                 bool to check
    println("Key exists")
} else {
    println("Key doesn't exist")
}

for

// Infinite loop
for {
    println("Works like `while true {}`")
    break
}

// Conditional `for` == while
for i < 5 {
    println(i)
    i++
}

// Conditional `for` w/ `if`
for i < 10 {
    if i%2 == 0 {
        i++
        continue
    }
    println(i)
    i++
}


sl := []int{1, 3, 5}

// Standard `for`
for i := 0; i < len(sl); i++ {
    println("Iteration")
}

// `for` w/ `range` for slice
for index, value := range sl {
    println(index, "\b:", value)
}

// `for` w/ `range` for map
m := make(map[string]string)

m["firstName"] = "Oleg"
m["lastName"] = "Legun"

for key, value := range m {
    println(key, ":", value)
}

switch

We don't fall through cases in Go - don't need break.

m := make(map[string]string)
m["firstName"] = "Oleg"

switch m["firstName"] {
case "Oleg":
    println("Name is set to 'Oleg'")

case "Igor":
    println("Name is set to 'Igor'")
    if true {
        break               // exit switch
    }
    println("Some text")

case "":
    println("Name is empty")
    fallthrough             // fall to next case

default:
    println("Please, set a valid 'firstName'")
}

Check condition directly in case. Can substitute multiple if-else.

switch { // tagless switch
case m["firstName"] == "Oleg":
    println("First Name: Oleg")

case m["firstName"] == "":
    println("First Name is empty")
}

To break out from loop inside switch, we need to use label.

m := make(map[string]string)
m["firstName"] = "Oleg"

Loop:
    for key, val := range m {
        switch {
        case key == "firstName" && val == "Oleg":
            println("Breaking...")
            break Loop
        }
        println("Won't be outputted")
    }

Function

Go has no concept of default parameter values.

In Go (call-by-value language) the called function always receives
only a copy of an argument, not a reference to the original argument.

// Local function
func sum(a int,  b int) int {
    return a + b
}

// Sum is an exported function and must have a comment
func Sum(a int, b int) int {
    return a + b
}

// Short notation (same type)
func sum(a, b int) int {
    return a + b
}

// Any number of arguments of the same type
func sum(sl ...int) int { // sl - variadic parameter (== []int)
    var res int
    for i:= range sl {
        res += sl[i]
    }
    return res
}

// Named return variable => initialized w/ null value (ready to use)
func sum() (res int) {
    res = 1
    return
}

// Multiple return
func multiReturn() (int, error) {
    return 1, nil
}

// Multiple return w/ named variables
func multiReturn() (res int, err error) {
    res = 1
    err = fmt.Errorf("Error")
    return
}

Getter method should be called Property (upper case, exported), not
GetProperty. A setter function, if needed, will likely be called
SetProperty.

Anonymous Function

Function in Go is first class citizen. We can save pointer to a
function or pass it in the moment of creation.

f := func() func() {  // return func()
    return func() {
        println("success")
    }
}

pointer = f() // pointer to function
pointer()     // "success"
f()()         // "success"

Closure

func main() {
    closure()()
}

func closure() func() {
    a := 1
    return func() { // has a in closure
        print(a)
    }
}

Recursive Functions

Are good for processing data w/ recursive structure. Unefficient on
memory, each new call expands stack. Good w/ last call optimization.

Many programming language implementations use a fixed-size function
call stack; sizes from 64KB to 2MB are typical. Fixed-size stacks
impose a limit on the depth of recursion, so one must be careful to
avoid a stack overflow when traversing large data structures
recursively; fixed-size stacks may even pose a security risk. In
contrast, typical Go implementations use variable-size stacks that
start small and grow as needed up to a limit on the order of a
gigabyte. This lets us use recursion safely and without worrying about
overflow.

defer

Block defer will be executed on exiting current function. The
canonical examples are unlocking a mutex or closing a file/connection.

Go's gc won't release unused OS resources like open files and
network connections. They must be closed explicitly.

The last defined defer will be executed first and so on (LIFO).

func main() {
    defer println("Done")
    println("Last command")
}

func ReadFile(f string) ([]byte, error) {
    file, err := os.OpenFile(...)
    if err != nil {
        return nil, err
    }
    defer file.Close()  // Now we can exit function any time

    return nil, nil
}

Right place for a defer statement that releases a resource is
immediately after the resource has been successfully acquired.

In defer obj.method().close() - .method() will be called right
after defer definition! Only .close() will be deferred. Arguments
to deferred functions are evaluated when the defer executes.

panic()

panic() - creates exception. We can catch it using recover(). Create
and handle panic - expensive operations!

Use panic when some impossible situation happens.

func main() {
	createPanic()
	println("Continue working")
}

func createPanic() {
	defer func() {
		// Handle panic
		if r := recover(); r != nil {
			println("Error happened")
		}

		// Capture the stack trace
		buf := make([]byte, 10000)
		runtime.Stack(buf, false)
		println("Stack:", string(buf))
	}()

	panic("Error") // Will immediately exit current function
}

init()

In every package function init() will be called at the start of
execution, before main().

Doesn't take any arguments or return any values.

package main

var user = os.Getenv("USER")
var a int

func init() {
    if user == "" {
        panic("no value for $USER")
    }
    a = 100  // Is set before main() executes
}

func main() {
    println(a)
}

It's used to initialize variables w/ needed values or to verify/repair
the program state before real execution begins.

All init() functions in all packages will be called simultaneously
=> order is unpredictable.

Pointers

Variable - storage which contains value. Pointer allows access/update
the value indirectly: w/o knowing the name of the variable.

a := 1
aPtr := &a  // *int
*aPtr = 10

bPtr := new(int) // return pointer to int
*bPtr = 10

Not every value has an address, but every variable does.

Each component of a variable of aggregate type (field of a struct,
element of an array) - is also a variable and thus han an address.

It is safe for a function to return the address of a local variable.

new vs. make

  • new() - allocates zeroed storage for any type T and returns
    pointer. new(type) is just &type{}. Use new when there is no
    need of a dummy name.
  • make() - only for slices, maps and channels, returns an initialized
    (not zeroed) value of type T (not *T)


Slice, map or channel represent references to data structures that
must be initialized using make before use. Until these
structures are initialized these types are nil. new only allocates
storage!

The new function is rarely used because the most common unnamed
variables are of struct types, for which the struct literal syntax
(User{}) is more flexible.

Heap vs. Stack

A compiler may choose to allocate local variables on the heap or on the
stack based on escape analysis. Each variable that escapes requires an
extra memory allocation.

var global *int

func f() {
    var x int
    x = 1
    global = &x // x escapes from f => stack allocation is not safe => allocate on heap
}

Named basic types

The named type provides a way to separate different uses of the
underlying type so that they can't be mixed unintentionally.

// Distinguishing the types makes it possible to avoid errors
// like combining temperatures in the two different scales.
type Celsius float64
type Fahrenheit float64

// Explicit conversion is required
tc := Celsius(-273.15)
tf := Fahrenheit(180)

// Error: mismatched types
temp := tf + tc

Struct (concrete type)

Type is a pointer to a struct.

Struct is a value type and describes properties of an object.

If variable is a large struct and performance is an issue, it's
preferable to pass variable by pointer *T. So that to avoid
expensive copying of the whole struct in memory.

We can compare structs (only for equality): they are equal if all
their properties are equal.

All fields in a struct are initialized w/ default nil values when
using T{}

type Text string

type Person struct {
    Name    string
    age     int
}

// Initialize w/ default values
var article Text  //   ""
var p1 Person     //  {"", 0}
p2 := Person{}    //  {"", 0}
p3 := new(Person) // &{"", 0}

// Initialize w/ custom values using literal
article := Text("Simple Text")
p := Person{
    Name:   "Oleg",  // Do not need to set all fields
    age:    26
}

// Short variant
p := Person{"Oleg", 26}  // Need to specify all fields

// Get pointer to a struct's field
name := &p.Name
*name = "Jack"

Struct can't contain itself (recursion), but it can contain pointer to
itself.

type tree struct { // binary tree
    value       int
    left, right *tree
}

Type methods

Any type like string, struct, func... can have its own methods. Go
doesn't use this or self, it uses receiver name.

type MyType int

// Value method uses copy of a type
func (m MyType) getInfo() {  // get m by value (copy)
    fmt.Printf("%T %v", m, m)
}

// Pointer method uses address of a type
func (m *MyType) Increment() {  // get m by reference (can change value)
    *m = *m + 1                 // Need to dereference pointer
}

Value methods can be invoked on pointers and values, but
pointer methods can only be invoked on pointers. This rule
arises because pointer methods can modify the receiver, invoking them on
a value would cause the method to receive a copy of the value, so any
modifications would be discarded.

// Person - simple type
type Person struct {
	name, surname   string
	age             int
}

// Method for type Person
func (p Person) getAge() int {
	return p.age
}

// Method for pointer to type Person
func (p *Person) setAge(age int) {
	p.age = age
}

func main() {
	oleg := Person{"Oleg", "Legun", 25}

	oleg.setAge(26)  // implicit &oleg on variable
	Person{"Jack", "", 10}.setAge(20) // compile error: can't take address of Person literal
	println(oleg.getAge())
}

If any method of a type T has a pointer receiver, you should avoid
copying instances of T because doing so may violate internal
invariants. For example, copying an instance of bytes.Buffer would
cause the original and the copy to alias the same underlying array of
bytes. Subsequent method calls would have unpredictable effects.

Method Values and Expressions

type User struct{ name string }

func (u *User) SetName(n string) { u.name = n }
func (u User) PrintName()        { fmt.Println(u.name) }

func main() {
    u := User{}

    setName := u.SetName           // Method value
    setName("Jack")
    PrintNameExp := User.PrintName // Method expression
    PrintNameExp(u)                // first arg == instance of User, then any args for method
}

Interface (abstract type)

Interface - describes behaviour = only methods that will be
implemented by some type.

One-method interfaces are named by the method name plus an -er
suffix: Sender, Reader, Closer...

type Speaker interface {
	Speak()
}

type Person struct {
	Name string
}

func (p Person) Speak() {
	println(p.Name + " is speaking...")
}

func SpeakTwice(s Speaker) { // Receives object w/ methods described in Speaker
	s.Speak()
	s.Speak()
}

func main() {
	oleg := Person{"Oleg"}
	oleg.Speak()
	SpeakTwice(oleg)
}

We can use Duck Typing in Go - if type struct implements all methods
from interface - we can pass it like this interface.

Use interface only when it's absolutely needed.

Empty interface as a type

interface{} - empty interface has no methods -> any type conforms to
it.

func Do(any interface{}) interface{} {
    // to use `any` we need to implement type assertion.
}

Interface Value

Interface value has 2 components:

  • concrete type (dynamic type)
  • value of its type (dynamic value)

Zero value for an interface has both its type and value components set
to nil.

var w io.Writer
/*  w
    type  =  nil
    value =  nil
*/
w.Write([]byte("text")) // panic: nil pointer dereference

w = os.Stdout
/*  w
    type  =  *os.File
    value =  *---------> os.File(fd int = 1(stdout))
*/
// dynamic value holds a copy of os.Stdout, which is a pointer to the
// os.File variable representing the standard output of the process
w.Write([]byte("text")) // success

We cannot know at compile time what the dynamic type of an
interface value will be, so a call through an interface must use
dynamic dispatch. Instead of a direct call, the compiler must
generate code to obtain the address of the method named Write from
the type descriptor, then make an indirect call to that address. The
receiver argument for the call is a copy of the interface’s dynamic
value, os.Stdout.

Interface with nil Pointer

func main() {
    var buf *bytes.Buffer  // must be io.Writer
    f(buf)
}

func f(out io.Writer) {
    if out != nil {                // dynamic value = nil, type = *bytes.Buffer, thus is not nil!
        out.Write([]byte("text"))  // panic: nil pointer dereference
    }
}

Embed

Similar to inheritance.

By embedding a struct we get its properties + methods.

type Person struct {
	name string
	Age  int
}

type Student struct {
	Person     // indicate only type
	course int
	Age    int // shadow property from Person
}

Only interfaces can be embedded within interfaces.

Error

func Do() err {
    return errors.New("simple error")
}

Any obj that implements error interface can be returned as an error.

type error interface {
    Error() string
}

Thus we can create custom error types.

// Create a named type for our new error type
type myError string

// Implement the error interface
func (e myError) Error() string {
       return string(e)  // wrapper above string
}

Message strings should not be capitalized and newlines should be
avoided.

Error Handling And Go

Type Assertion

having created an interface{} value containing a boolean,
float, string, map, pointer, or any other type, we can
do nothing directly to the value it holds since the interface has
no methods. We need a way to get the value back out again.

func DoubleSpeak(s interface{}) { // Receives object w/ methods described in Speaker
    if p, ok := s.(Speaker); ok { // if s implements Speaker interface
        p.Speak()
        p.Speak()
    }
    if p, ok := s.(Person); ok {  // if s obj is of type Person
        println(p.Name)           // use property of p
    }
}

Switch Type Assertion

Check type:

func showMeTheType(i interface{}) string {
   switch i.(type) {
   case int:
      return "int"
   case uint:
      return "uint"
   case int8:
      return "int8"
   case float64:
      return "float64"
   case string:
      return "string"
   case rune:
      return "int32"
   case []int:
      return "[]int"
   case map[string]bool:
      return "map[string]bool"
   default:
      return "undefined"
   }
}

err.(type) works only in switch.

Check error type:

switch err := err.(type) {  // Check error type
    case nil:

    case *MyError:

    default:
}

Goroutines

Overview

Go routine - "simplified" thread, independent code that executes
asynchronously and has its own stack. To start go routine we need 2-4KB
(depending on the architecture).

Process - in OS, have separate address space (independent). We need to
use pipes or other OS mechanisms to coordinate processes.

Threads are within Process. Have common assets - can conflict (use
mutex). To start thread we need 1MB for its data.

In Go we don't know what thread we are working on. All concurrency is
managed by Go runtime.

Go runtime manages dispatching go routines on threads.

Go routines can lead to memory leaks, if not finished correctly.

Concurrency — structuring a program as independently executing
components and parallelism — executing calculations in parallel for
efficiency on multiple CPUs.

package main

import "fmt"

func main() {
   go test() // tells runtime to create "subprocess" w/ its own stack

   go print(5) // go routine w/ parameters

   go func() { // anonymous
      println("anonymous go routine")
   }()

   fmt.Scanln() // We need to wait go routine completion
}

func test() {
   println("routine")
}

func print(i int) {
   println(i)
}

go run -race

Used to debug concurrent programs.

Go routines that operate on the same object can cause unexpected changes
(they need to be synced). Check problems w/ go run -race .... -race
arbitrary switches between goroutines in runtime.

go run -race main.go
...
Found 5 data race(s)
exit status 66

Indicates that 5 simultaneous read/write operations happened w/o blocks.

go routine & closure

for i := 0; i < 5; i++ {
    go func() {
        println(i)  // will print '4' every time
    }()
}

When async println(i) is invoked - for loop has already finished =>
i == 4

We need to explicitly pass needed values to save them in closure:

for i := 0; i < 5; i++ {
    go func(i int) {
        println(i)  // will print 1 2 3 4
    }(i)
}

Shadowing variable

It's legal and idiomatic in Go to do this. You get a fresh version of
the variable with the same name, deliberately shadowing the loop
variable locally but unique to each goroutine.

func Serve(queue chan *Request) {
    for req := range queue {
        req := req // Create new instance of req for the goroutine.
        sem <- 1
        go func() {
            process(req)
            <-sem
        }()
    }
}

Channel

Overview

Channels are pipes for concurrent go routines. We can read from/write to
channel and it works in sync.

Channel is a reference type (pointer to a struct inside).

Buffered & unbuffered channels

Channel can be unbuffered or buffered.

Unbuffered channel works in synchronous mode - we can't write to
channel, unless somebody is ready to read from it.

func main() {
   ch := make(chan string) // Create channel for strings

   go writeToChannel(ch) // Asynchronously write to channel

   fmt.Println(<-ch, ",", <-ch) // Read 2 times from channel
}

func writeToChannel(ch chan<- string) { // receive channel (implicitly indicate write-only channel)
   ch <- fmt.Sprintf("Foo") // will pause until smb. is ready to read from ch
   ch <- fmt.Sprintf("Bar")
   ch <- fmt.Sprintf("Baz") // will not be printed, since we read only 2 times from ch
}

Channel direction (prevents errors in compile time - it's just an
instruction for compiler checks):

  • <-chan - read-only
  • chan<- - write-only

To use buffered channel, we must implicitly indicate its size.

bufCh := make(chan int, 7) // Create buffered channel for ints

We can write to bufCh 7 times before our program execution blocks (in
case nobody read from bufCh).

To close channel:

close(bufCh)

// Check if buffer is closed
data, ok := <-bufCh // if opened: ok == true, if closed -> false

We can't write to closed channel (panic), but can read from it (always
returns nil)

Read from channel w/ range

func readFromChannel(ch <-chan string) {
    for r := range ch {
        println(r)
    }
}

If chanel is closed - range will stop iterating.

Channel generator

func main() {
   channel := createChannel("Hello")

   for {
      fmt.Println(<-channel)
   }
}

func createChannel(message string) <-chan string {
   ch := make(chan string)

   go func() {
      for i := 0; ; i++ {  // Infinite loop
         ch <- fmt.Sprintf("%d. %s", i, message)
         time.Sleep(time.Second)
      }
   }()

   return ch
}

time.After(interval) - is a channel generator, that returns channel
that receives bool values true each time interval passes.

Sync go routines using channels

We can use chan struct{} or chan bool to implement done channel.
struct{} has zero size, but syntax is more cumbersome ch <- struct{}{} and space saving is marginal.

func main() {
   done := make(chan bool, 1)
   go mustPrint(done)

   // Blocking receive
   <-done // await, read from done, otherwise program will exit earlier
}

func mustPrint(done chan<- bool) {
   for i := 0; i < 10; i++ {
      println(i)
   }

   done <- true // write to done on finish
}

Channel multiplexer

Read several channels simultaneously. Order is not guaranteed, i.e.
working w/ unsynced channels.

func main() {
   ch1, ch2 := generateNumbers()
   ch := readAll(ch1, ch2)

   for {
      fmt.Println(<-ch)
      time.Sleep(time.Second)
   }
}

func generateNumbers() (<-chan int, <-chan int) {
   ch1, ch2 := make(chan int, 1), make(chan int, 1)

   go func() {
      for i := 0; i < 10; i++ {
         ch1 <- i
      }
   }()

   go func() {
      for i := 10; i > 0; i-- {
         ch2 <- i
      }
   }()

   return ch1, ch2
}

// readAll reads 2 channels simultaneously and writes result to the returned channel
func readAll(ch1, ch2 <-chan int) <-chan int {
   ch := make(chan int)

   go func() {
      for {
         ch <- <-ch1
      }
   }()

   go func() {
      for {
         ch <- <-ch2
      }
   }()

   return ch
}

Select

Works like switch but for channels.

Checks all cases in order and checks for available data in channels. If
any - case will be executed, if not - waits and passes control to other
routines.

for {
   select {
   case msg := <-ch1:
      println(msg)
   case msg := <-ch2:
      println(msg)
   case <-time.After(time.Second): // Guarantees non-blocking execution
      println("timeout")  // excellent for networking requests
   }
   runtime.Gosched()  // Runtime can pass control to other routine
}

If select doesn't have default case - program blocks execution
and waits for input on any channel, otherwise it executes default
case.

select {
default:  // non-blocking channel operation
    println("default")
}

default case guarantees infinite loop - no code will be blocked if
no channel is ready for read.

Priority select: 1 channel is more important that another.

Go runtime can't take execution control from the subroutine in arbitrary
moment (when it's rendering smth.), but runtime.Gosched() tells
runtime that there will be waiting, and it can pass control to other
routine during it. Useful in infinite loops!

select can also act as a channel multiplexer:

func readAll(ch1, ch2 <-chan int) <-chan int {
    ch := make(chan int)

    go func() {
        for {
            select {
            case num := <-ch1:
                ch <- num
            case num := <-ch2:
                ch <- num
            }
        }
    }()
}

Packages

io

Working with byte streams.

Reader

type Reader interface {
    // Pass buffer p to reuse the same bytes (helps GC)
    // p is not guaranteed to be filled: 0 - len(p)
    // returns io.EOF on stream done (normal behavior)
    Read(p []byte) (n int, err error)
}

ReadFull(), ReadAtLeast()

// ReadFull ensures that buf is completely filled
// Returns io.EOF or io.ErrUnexpectedEOF on partial read
func ReadFull(r Reader, buf []byte) (n int, err error)
// ReadAtLeast reads at least min bytes
// We can minimize a number of Read() calls buffering additional data
func ReadAtLeast(r Reader, buf []byte, min int) (n int, err error)

MultiReader()

// MultiReader combines n Readers into one concatenated reader
func MultiReader(readers ...Reader) Reader
r := io.MultiReader(
    bytes.NewReader([]byte("string")),
    file,
)
http.Post("http://...", "application/octet-stream", r)

TeeReader()

// TeeReader duplicates streams: returns new Reader, that wraps r
// Any reads from r will be written to w (ex. bytes.Buffer{}) for further inspection
func TeeReader(r Reader, w Writer) Reader

LimitReader()

// LimitReader wraps r and stops reading with EOF after n bytes or less
func LimitReader(r Reader, n int64) Reader

Set n = n + 1 to check if r has more than n bytes to read

Writer

type Writer interface {
    Write(p []byte) (n int, err error)
}

MultiWriter()

// MultiWriter duplicates writes to multiple streams (useful for logging)
func MultiWriter(writers ...Writer) Writer

WriteString()

// WriteString uses optimized WriteString() method of w if available
// Optimization: converts string to a []byte w/o allocations
func WriteString(w Writer, s string) (n int, err error)

Copy(), CopyN(), CopyBuffer()

// Copy copies all bytes from Reader to Writer using 32KB buffer
func Copy(dst Writer, src Reader) (written int64, err error)
// CopyN copies N bytes from Reader to Writer
func CopyN(dst Writer, src Reader, n int64) (written int64, err error)
// CopyBuffer uses passed buf []byte w/o allocation its own 32KB buffer
func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error)

WriterTo, ReaderFrom

type WriterTo interface {
    // Directly write data to w w/o intermediate buffer (optimized)
    WriteTo(w Writer) (n int64, err error)
}

type ReaderFrom interface {
    ReadFrom(r Reader) (n int64, err error)
}

Pipe()

// Pipe provides new Reader and Writer, any writes to Writer will go to Reader
// Useful if we have a method that only accepts a Reader, but we only have a Writer
func Pipe() (*PipeReader, *PipeWriter)

Closer

type Closer interface {
    Close() error
}

Seeker

type Seeker interface {
    // Seek is useful for fixed length blocks in a file
    // whence can be io.SeekStart, io.SeekCurrent, io.SeekEnd
    Seek(offset int64, whence int) (int 64, error)
}

ByteReader, ByteWriter, ByteScanner

type ByteReader interface {
    // Read single byte
    ReadByte() (c byte, err error)
}

type ByteWriter interface {
    WriteByte(c byte) error
}

// For working with buffered byte readers
// that push the prev byte back for reading (peek next byte)
type ByteScanner interface {
    ByteReader
    UnreadByte() error
}

RuneReader, RuneScanner

type RuneReader interface {
    ReadRune() (r rune, size int, err error)
}

type RuneScanner interface {
    RuneReader
    UnreadRune() error
}

bytes, strings

Working with bounded, in-memory byte slices. The elements of a byte
slice can be freely modified.

  • []byte - mutable, resizable, contiguous list of bytes
  • string - immutable, fixed-size, contiguous list of bytes.

NewReader

Create Reader that wraps an in-memory []byte or string. It also
implements io interfaces: WriterTo, ByteReader, ByteScanner,
RuneReader, RuneScanner, Seeker.

func NewReader(b []byte) *Reader
func NewReader(s string) *Reader
// Effective on memory
r := strings.NewReader("oleg")  // Can be inside of io.MultiReader() if needed
http.Post("http://...", "text/plain", r)

Buffer

For buffer use bufio package.

// Buffer of bytes with many Read and Write methods.
// The zero value for Buffer is an empty buffer ready to use.
type Buffer struct { /* filtered */ }

// Useful Type Methods
buf.WriteString(s string) (n int, err error) {} // appends string to the end of the buffer
buf.Reset() // Empties buffer
var buf bytes.Buffer
myApp.Logger = log.New(&buf, "", log.LstdFlags)
myAPp.Start()

if !strings.Contains(buf.String(), "app failed") {
        log.Fatal("Message")
}
func Equal(a, b []byte) bool {}     // check for equality byte slices, for strings use ==
func EqualFold(s, t string) bool {} // check for equality 2 string ignoring case

func Contains(b, subslice []byte) bool {}// 1+ subslice in b
func Contains(s, substr string) bool {}  // 1+ substr in s
func Count(b, subslice []byte) int {}    // Exact num of sublices in b
func Count(s, substr string) int {}      // Exact num of substr in s, if substr is "" -> returns len+1 of chars

// Find position
func Index(s, sep []byte) int {}
func IndexAny(s []byte, chars string) int {}
func IndexByte(s []byte, c byte) int {}
func IndexFunc(s []byte, f func(r rune) bool {}) int {}
func IndexRune(s []byte, r rune) int {}

// Includes validation for blank slice (no panic)
func HasPrefix(s, prefix []byte) bool {}  // "PrefixString"
func HasPrefix(s, prefix string) bool {}
func HasSuffix(s, suffix []byte) bool {}  // "StringSuffix"
func HasSuffix(s, suffix string) bool {}

// Remove any runes defines in cutset from the beginning/end
func Trim(s []byte, cutset string) []byte {}
func Trim(s string, cutset string) string {}
func TrimLeft(s []byte, cutset string) []byte {}
func TrimRight(s []byte, cutset string) []byte {}
func TrimFunc(s []byte, f func(r rune) bool {}) []byte {}

// Remove prefix/suffix
func TrimPrefix(s, prefix []byte) []byte {}
func TrimPrefix(s, prefix string) string {}
func TrimSuffix(s, suffix []byte) []byte {}
func TrimSuffix(s, suffix string) string {}

// Remove space from the beginning/end
func TrimSpace(s []byte) []byte {}
func TrimSpace(s string) string {}

// Replace (for templates)
func Replace(s, old, new []byte, n int) []byte {}
func Replace(s, old, new string, n int) string {}

// Change case
func ToUpper(s []byte) []byte {}
func ToUpper(s string) string {}
func ToLower(s []byte) []byte {}
func ToLower(s string) string {}
func Title(s []byte) []byte {}
func Title(s string) string {} // "hello world" -> "Hello World"

// Split slice using separator
func Split(s, sep []byte) [][]byte {}
func Split(s, sep string) []string {} // ("1:2:3", ":") -> ["1", "2", "3"]
func SplitN(s, sep []byte, n int) [][]byte {}
func SplitN(s, sep string, n int) []string {}
func SplitAfter(s, sep []byte) [][]byte {}
func SplitAfter(s, sep string) []string {}
func SplitAfterN(s, sep []byte, n int) [][]byte {}
func SplitAfterN(s, sep string, n int) []string {}
func Fields(s string) []string {}         // ("hello    world") -> ["hello", "world"]

func Join(s [][]byte, sep []byte) []byte {}
func Join(a []string, sep string) string {}
func Repeat(s string, count int) string {}  // ("-", 5) -> "-----"

bufio

// Scanner reads input and breaks it into lines/words.
// Best way to process input that comes in lines
type Scanner struct {}

input := bufio.NewScanner(os.Stdin)
input.Scan() // reads next line + removes `\n`; returns false
input.Text() // retrieve result from scan

encoding

  • Encoding/Decoding applies a logical structure to a stream of
    bytes (io.Reader/io.Writer)
  • Marshaling/Unmarshaling applies a logical structure to bounded,
    in-memory bytes ([]byte)

Go structures are already represented in-memory as bytes. We can use
them directly. gob (stream encoding) included the schema format when
encoding, but is large. No cross language support.

type BinaryMarshaler interface {
    MarshalBinary() (data []byte, err error)
}
type BinaryUnmarshaler interface {
    UnmarshalBinary(data []byte) error
}

// Output in UTF-8 format
type TextMarshaler interface {
    MarshalText() (text []byte, err error)
}
type TextUnmarshaler interface {
    UnmarshalText(text []byte) error
}

encoding/json

JSON encoding/decoding is slow. Encoder checks for a MarshalJSON
method of an type, then for MarshalText. If no one method is
implemented - it will recursively build an encoder based on the
primitive encoders (intEncoder, stringEncoder, mapEncoder,
structEncoder, sliceEncoder).

type Marshaler interface {
    MarshalJSON() ([]byte, error)
}
type Unmarshaler interface {
    UnmarshalJSON([]byte) error
}

Channel, complex types, functions cannot be encoded in JSON.

RawMessage

RawMessage is a raw encoded JSON value. It implements Marshaler and
Unmarshaler and can be used to delay JSON decoding or precompute a
JSON encoding.

type T struct {
	Type  string          `json:"type"`
	Value json.RawMessage `json:"value"`
}

func (t *T) Val() (interface{}, error) {
    switch t.Type {
    case "foo":
        // parse "t.Value" as Foo
    case "bar":
        // parse "t.Value" as Bar
    default:
        return nil, errors.New("invalid type")
    }
}

Indent(), SetIndent(), Compact()

Pretty print JSON.

func Indent(dst *bytes.Buffer, src []byte, prefix, indent string) error {}
func Compact(dst *bytes.Buffer, src []byte) error {}

// For using with Encoder
func (enc *Encoder) SetIndent(prefix, indent string) {}

JSON package has many error types, there’s not much we can do to handle
them in code other than log an error and have a human operator
intervene.

ffjson lib - faster JSON processing for Go

NewEncoder(), NewDecoder()

Useful for efficient encoding/decoding with less heap allocations.

type Person struct {
    FirstName string `json:"first_name"`
    LastName  string `json:"last_name"`
}

func main() {
    person := Person{"Oleg", "Legun"}

    // Marshal - save to []byte
    personEncoded, err := json.Marshal(person)
    if err != nil {
        log.Fatal(err)
    }

    personDecoded := Person{}

    // Unmarshal
    err = json.Unmarshal(personEncoded, &personDecoded)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Unmarshal: %+v\n", personDecoded)

    // Encoder - encode & write strait to the Writer
    encoder := json.NewEncoder(os.Stdout)
    err = encoder.Encode(person)
    if err != nil {
        log.Fatal(err)
    }

    // Decoder - read & decode strait from the Reader
    reader := bytes.NewReader(personEncoded)
    decoder := json.NewDecoder(reader)
    err = decoder.Decode(&personDecoded)
    if err != nil {
        log.Fatal(err)
    }
}

Struct Field Attributes

type Person struct {
    // Change the field name in JSON output
    FirstName  string `json:"first_name"`
    // Lowercase (unexported) properties are ignored by Marshal()
    lastName   string `json:"last_name"`
    // Omit this field in output
    Patronymic string `json"-"`
    // Omit this non-struct field if empty
    Age        int    `json:"age, omitempty"`
    // Convert field value to a quoted string (very important for 64-bit values
    // JS has problems with values above 53 bits start to lose integer precision
    Phone      int    `json:",string"`
}

json.Encoder uses sync.Pool to reuse internal buffer (minimize heap
allocations).

encoding/binary

  • In text protocol like JSON number 128 is represented with 1 + 2 + 8
    chars.
  • In binary protocol 128 is represented with a single byte 0x80

Endianness

  • Little-endian (CPU byte order = resizable) - [less signif. byte][most signif. byte] = 11 22 33 (00 00)
  • Big endian (network byte order = sortable) - [most signif. byte][less signif. byte] = 33 22 11


v := uint32(65534)
buf := make([]byte, 4)

binary.BigEndian.PutUint32(buf, v)    // [00 00 ff fe]
binary.LittleEndian.PutUint32(buf, v) // [fe ff 00 00]

Variable-length encoding

Small numbers take less space (trim empty zero bytes).

// Encode varint x to a []byte slice
func PutUvarint(buf []byte, x uint64) int {}
func PutVarint(buf []byte, x int64) int {}

// Decode varint from a []byte slice
func Uvarint(buf []byte) (uint64, int) {}
func Varint(buf []byte) (int64, int) {}

// Decode varint from a byte stream (one by one)
func ReadUvarint(r io.ByteReader) (uint64, error) {}
func ReadVarint(r io.ByteReader) (int64, error) {}

strconv

Package for efficient conversions of primitive values (bool, int,
float, string). fmt is slower.

int

// Parse int from a string
// base: 2-36 (0 - auto); bitSize: 8-64 (0 - default system int size 32/64)
func ParseInt(s string, base int, bitSize int) (int64, error) {}
func ParseUint(s string, base int, bitSize int) (uint64, error) {}

func Atoi(s string) (int, error) {} // == ParseInt(s, 10 ,0)

// int -> string (base: 2 - 36 [0-9 + a-z])
func FormatInt(i int64, base int) string {}  // new string is allocated on each call
func FormatUint(i uint64, base int) string {}

// int -> string (less allocations, reusable buffer)
func AppendInt(dst []byte, i int64, base int) []byte {}
func AppendUint(dst []byte, i uint64, base int) []byte {}

float

// Parse float from a string, bitSize: 32/64
func ParseFloat(s string, bitSize int) (float64, error) {}

// float -> string
// prec: -1 - auto from bitSize (smallest num possible), bitSize: 32/64
func FormatFloat(f float64, fmt byte, prec, bitSize int) string {}

boolean

// true:  "1" "t" "T" "true" "True" "TRUE"
// false: "0" "f" "F" "false" "False" "FALSE"
// error: others
func ParseBool(str string) (bool, error) {}

// bool -> string; returns "true"/"false"
func FormatBool(b bool) string {}

// bool -> string
// Appends "true"/"false" to the end of dst, returns extended buffer
func AppendBool(dst []byte, b bool) []byte {}

string

// Quote (double quotes) string s, converting non-printable chars to `\t`, `\n`, `uXXXX`
// Useful for displaying error messages.
func Quote(s string) string {} // "Hello   " -> "\"Hello\t\""

// Converts all non ASCII chars (non-printable, emoji)
func QuoteToASCII(s string) string {}

// Less heap allocations when quoting strings
func AppendQuote(dst []byte, s string) []byte {}
func AppendQuoteToASCII(dst []byte, s string) []byte {}


// Quote individual rune (single quotes)
func QuoteRune(r rune) string {}
func QuoteRuneToASCII(r rune) string {}

// Less heap allocations when quoting runes
func AppendQuoteRune(dst []byte, r rune) []byte {}
func AppendQuoteRuneToASCII(dst []byte, r rune) []byte {}


// Unquote strings (parses "", '' and `` strings)
func Unquote(s string) (string, error) {}

fmt

%v    default format
      bool:                    %t
      int, int8 etc.:          %d
      uint, uint8 etc.:        %d, %#x if printed with %#v
      float32, complex64, etc: %g
      string:                  %s
      chan:                    %p
      pointer:                 %p
%+v   add field names
%#v   valid Go syntax, can be pasted into the code right away
%T    print type (useful for the dynamic type of an interface value)
%d    int (base-10)
%x    int (base-16)
%f    float
       %5.2f   1.23 -> "  1.23" (width = 5, precision = 2)
       %-5.2f  1.23 -> "1.23  "
       %05.2f  1.23 -> "001.23"

%q    quoted string (use-case: print with invisible chars)
%p    pointer (use-case: check if 2 pointer variables reference the same data)
// Print to os.Stdout
func Print(a ...interface{}) (n int, err error) {}
func Printf(format string, a ...interface{}) (n int, err error) {}
func Println(a ...interface{}) (n int, err error) {}

// Print to io.Writer, F stands for File
func Fprint(w io.Writer, a ...interface{}) (n int, err error) {}
func Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error) {}
func Fprintln(w io.Writer, a ...interface{}) (n int, err error) {}

LogOutput := os.Stdout
fmt.FPrintln(LogOutput, "Error")

// Print to string (slow! - use reusable bytes.Buffer with Fprint())
func Sprint(a ...interface{}) string {}
func Sprintf(format string, a ...interface{}) string {}
func Sprintln(a ...interface{}) string {}

// Just a wrapper for errors.New(Sprintf())
func Errorf(format string, a ...interface{}) error {}

// Use for custom types
type Stringer interface {
    String() string
}

Do not use Scan functions - use CLI flags, env vars, API calls
instead, config files.

http

ServeMux

HTTP request multiplexer. Matches URLs of incoming requests against a
list of registered patterns and calls the handler for the pattern that
most closely matches the URL. A ServeMux aggregates a collection of
http.Handlers into a single http.Handler. Doesn't support wildcards
and RegExp.

Matching rules:

  • / matches all unmatched paths + root
  • long path has more priority over short path (/src/img/ >
    /src/)
  • /src will be redirected to /src/ if handler for /src/ is
    registered (override - register handler for /src)

Several ServeMux may be composed to handle more intricate
dispatching requirements.

Handler, HandlerFunc, Handle(), HandleFunc()

// Any object that implement Handler can used as a router
type Handler interface {
    // Return from ServeHTTP == request is finished
    ServeHTTP(ResponseWriter, *Request)
}

// HandlerFunc is an Adapter to allow the use of functions as HTTP handlers
// its ServeHTTP() method calls underlying function
type HandlerFunc func(ResponseWriter, *Request)

To add handlers to a ServeMux use:

// DefaultServeMux

// Requires an object that implements Handler interface
func Handle(pattern string, handler Handler) {}


// HandleFunc registers a handler function for the given pattern
// ServeHTTP() method calls handler() passing (ResponseWriter, *Request)
func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {}
// Custom ServeMux
mux := http.NewServeMux()
mux.Handle("/", http.HandlerFunc(f))
mux.HandleFunc("/", f) // the same
log.Fatal(http.ListenAndServe("localhost:8080", mux)

func f(w http.ResponseWriter, r *http.Request) {/* ... */}

ListenAndServe()

func ListenAndServe(addr string, handler Handler) error {}

// if handler == nil, use DefaultServeMux (instance of ServeMux)
http.ListenAndServe(":8080", nil)

Handlers

Go web server invokes each handler in a new goroutine, so handlers
must take precautions such as locking when accessing variables the
other goroutines may be accessing.

// FileServer returns Handler for serving static files
func FileServer(root FileSystem) Handler {}

http.Handle("/images", http.FileServer(http.Dir("./images")))


// NotFoundHandler returns Handler that replies to each request with a 404 error
func NotFoundHandler() Handler {}


// RedirectHandler returns Handler that redirects each request to url using status code
// Codes: http.StatusMovedPermanently, http.StatusFound, http.StatusSeeOther
func RedirectHandler(url string, code int) Handler {}


// StripPrefix returns Handler removes given prefix from a request and passes it to h
func StripPrefix(prefix string, h Handler) Handler {}

http.Handle("/files/", http.StripPrefix("/files/", http.FileServer(http.Dir("./"))))


// TimeoutHandler returns Handler that runs h with the given time limit
// On timeout handler responds with a http.StatusServiceUnavailable and msg in body
func TimeoutHandler(h Handler, dt time.Duration, msg string) Handler {}

Chain Handlers (middleware)

Useful for authentication.

type clientRequest struct {
    Data string `json:"data"`
}

type validationHandler struct {
    next http.Handler
}

func newValidationHandler(next http.Handler) http.Handler {
    return validationHandler{next: next}
}

func (v validationHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
    var request clientRequest

    decoder := json.NewDecoder(r.Body)
    err := decoder.Decode(&request)
    if err != nil {
        http.Error(rw, "Bad Request", http.StatusBadRequest)
        return // stop handler chaining on error
    }

    v.next.ServeHTTP(rw, r)
}

func main() {
    handler := newValidationHandler(http.NotFoundHandler())

    http.Handle("/", handler)
    http.ListenAndServe(":8080", nil)
}

Get()/Post()

// Get follows the redirect, up to a maximum of 10 redirects.
func Get(url string) (resp *Response, err error)

resp, err := http.Get(url)
b, err := ioutil.ReadAll(resp.Body)
resp.Body.Close() // close to avoid resource leaking

// To set custom headers, use NewRequest and DefaultClient.Do.
func Post(url string, contentType string, body io.Reader) (resp *Response, err error)

context

Useful for passing data from one handler to the next w/o breaking the
http.Handler interface.

// Returns an empty context (used as a top-level context by the main function)
func Background() Context {}

// Returns a copy of the parent Context with a cancel function; cancel() releases
// resources associated with the context (call after all operations are complete)
func WithCancel(parent Context) (ctx Context, cancel CancelFunc) {}

// Returns a copy of the parent context that expires after the current time > deadline
// CancelFunc - method for manual context cancellation
func WithDeadline(parent Context, deadline time.Time) (Context, CancelFunc) {}

// Similar to WithDeadline, except we pass it a duration for which Context should exist
func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc) {}

// Returns a copy of the parent context in which {key: val} is set
func WithValue(parent Context, key interface{}, val interface{}) Context {}

Context.Value(key interface{}) interface{}

For inbound requests http.Server automatically cancels context
when the client connection closes. For outbound requests Context()
controls cancellation.

// Inbound request

// Context often flows across packages, so we need to create package level type
// for use with context, otherwise collisions can happen with other keys (same name)
type validationContextKey string

func (h validationHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
    /* ... */
    // Set key in new context
    c := context.WithValue(r.Context(), validationContextKey("data"), request.Data)
    // Apply new context
    r = r.WithContext(c)

    // Pass the request with new context down the chain
    h.next.ServeHTTP(rw, r)
}

// Inside "next" handler:
// Retrieve value from context using key + type assertion
data := r.Context().Value(validationContextKey("data")).(string)
// Outbound request

r, err := http.NewRequest("GET", "http://site.com", nil)
if err != nil {
    log.Fatal(err)
}

// r.Context() - parent context
// timeoutRequest - new context with timeout
timeoutRequest, cancelFunc := context.WithTimeout(r.Context(), 1 * time.Millisecond)

// Cancel the Context() method of the outbound request == cancel the request itself!
defer cancelFunc()

// Apply new context to r (Request.WithContext(ctx context.Context) *Request)
r = r.WithContext(timeoutRequest)

_, err = http.DefaultClient.Do(r)
if err != nil {
    log.Fatal(err)
}

net/rpc

RPC Handler's methods must be:

  • the method's type is exported.
  • the method is exported.
  • the method has two arguments, both exported (or builtin) types.
  • the method's second argument is a pointer.
  • the method has return type error.

// Register (publish) methods of rcvr `Type.Method`
func Register(rcvr interface{}) error {}

// Register (publish) methods of rcvr with custom type name `CustomType.Method`
RegisterName(name string, rcvr interface{}) error {}

RPC over TCP

// contract/entities.go
type RPCHandlerRequest struct {
    A, B int
}

type RPCHandlerResponse struct {
    Result int
}


// server.go
type RPCHandler struct{}

// We do not need to conform to an interface, all methods of RPCHandler will be registered (published) in RPC server
func (h *RPCHandler) PublicMethodSum(args *contract.RPCHandlerRequest, reply *contract.RPCHandlerResponse) error {
    reply.Result = args.A + args.B
    return nil
}

func main() {
    err := rpc.Register(&RPCHandler{})
    FatalOnErr(err)

    // Create socket based on protocol binded with ip + port
    l, err := net.Listen("tcp", ":8080")
    FatalOnErr(err)

    for {
        conn, err := l.Accept()
        FatalOnErr(err)

        print("New connection!")
        // ServeConn() is a blocking method => using goroutine
        // ServeConn uses gob binary format
        go rpc.ServeConn(conn)
    }
}


// client.go
func main() {
    c, err := rpc.Dial("tcp", ":8080")
    if err != nil {
        log.Fatal(err)
    }
    args := contract.RPCHandlerRequest{2, 3}
    var reply RPCHandlerResponse
    err = c.Call("RPCHandler.PublicMethodSum", args, &reply)
    FatalOnErr(err)

    fmt.Printf("%+v", reply)
}

With gobs the source and destination values and types do not need to
correspond exactly. Non-existing field will be ignored (w/o error).

RPC over HTTP

// Sets up 2 endpoints to use HTTP as a transport:
// DefaultRPCPath = "/_goRPC_"
// DefaultDebugPath = "/debug/rpc"
func HandleHTTP() {}
// server.go
err := rpc.Register(&RPCHandler{})
FatalOnErr(err)
rpc.HandleHTTP()

l, err := net.Listen("tcp", ":8080")
FatalOnErr(err)

http.Serve(l, nil)


// client.go
client, err := rpc.DialHTTP("tcp", ":8080")

// sync function invocation
err = client.Call("RPCHandler.PublicMethodSum", args, &reply)
// async function invocation
call := client.Go("RPCHandler.PublicMethodSum", args, reply, nil)
replyCall := <-call.Done	// will be equal to call

With RPC over HTTP we can take advantage of HTTP headers and other
metadata

JSON-RPC over HTTP

net/rpc/jsonrpc provides a build-in codec for
serializing/deserializing to the JSON-RPC standard.

type ClientRequest struct {
    A, B int
}

type ServerResponse struct {
    Result int
}

type RPCHandler struct{}

func (h *RPCHandler) Sum(args *ClientRequest, reply *ServerResponse) error {
    reply.Result = args.A + args.B
    return nil
}

type HttpConn struct {
    in  io.Reader
    out io.Writer
}
// Methods to proxy calls to the reader and writer
func (c *HttpConn) Read(p []byte) (n int, err error)  { return c.in.Read(p) }
func (c *HttpConn) Write(d []byte) (n int, err error) { return c.out.Write(d) }
func (c *HttpConn) Close() error                      { return nil }

func main() {
    rpc.Register(new(RPCHandler))

    l, err := net.Listen("tcp", ":8080")
    if err != nil {
        log.Fatal(err)
    }

    http.Serve(l, http.HandlerFunc(httpHandler))
}

func httpHandler(w http.ResponseWriter, r *http.Request) {
    // Pass a type ReadWriteCloser
    serverCodec := jsonrpc.NewServerCodec(&HttpConn{in: r.Body, out: w})
    err := rpc.ServeRequest(serverCodec)
    if err != nil {
        log.Fatal(err)
    }
}

curl -X POST -H "Content-Type: application/json" -d '{"id": 1, "method": "RPCHandler.Sum", "params": [{"A":1,"B":2}]}' http://localhost:8080

net/http/httputil

// DumpRequest dumps HTTP request in ith HTTP/1.x representation into []byte
func DumpRequest(req *http.Request, body bool) ([]byte, error) {}

/* Result:

POST /api/v3/... HTTP/1.1
Host: 10.1.1.2:3000
Accept-Encoding: gzip
Content-Length: 100
Content-Type: application/json

POST_REQUEST_BODY_HERE
*/

html

// Parse returns the parse tree for the HTML from the given Reader
func Parse(r io.Reader) (*Node, error) {}

sort

// To sort any sequence we need to define a type that implements 3 methods
type Interface interface {
        Len() int
        Less(i, j int) bool // i, j - indices of sequence
        Swap(i, j int)
}

Sort(data Interface) {}

// Reverse - decorator, the returns Interface with inversed Less() method
ReverseSort(data Interface) Interface {}
// Makes an object sortable
type StringSlice []string
type IntSlice []int
type Float64Slice []float


func Strings(a []string) {} // Sorts a slice of strings in increasing order
type User struct {
    Name string
    Age  int
}

type byAge []*User

func (u byAge) Len() int           { return len(u) }
func (u byAge) Less(i, j int) bool { return u[i].Age < u[j].Age }
func (u byAge) Swap(i, j int)      { u[i], u[j] = u[j], u[i] }

type byName []*User

func (u byName) Len() int           { return len(u) }
func (u byName) Less(i, j int) bool { return u[i].Name < u[j].Name }
func (u byName) Swap(i, j int)      { u[i], u[j] = u[j], u[i] }


type customSort struct {
    u    []*User
    less func(x, y *User) bool
}

func (x customSort) Len() int           { return len(x.u) }
func (x customSort) Less(i, j int) bool { return x.less(x.u[i], x.u[j]) }
func (x customSort) Swap(i, j int)      { x.u[i], x.u[j] = x.u[j], x.u[i] }

func main() {
    users := []*User{{"Jack", 20}, {"Alice", 22}, {"Bob", 10}, {"Nick", 15}}

    sort.Sort(byAge(users))
    sort.Sort(sort.Reverse(byName(users)))
    sort.Sort(customSort{users, func(x, y *User) bool { return x.Name < y.Name }})
}

We use type byAge []*user instead of type byAge []user for faster
sort (sort swaps pointers instead of values).

compress/gzip

Use https://github.com/NYTimes/gziphandler for gzip.

ioutil

ioutil.Discard // io.Writer on which all Write calls succeed

sync

Low-level synchronization for go routines.

sync/atomic

Used for working w/ single number to guarantee atomic operations on it.

atomic.AddInt64(&num, 5)
atomic.LoadInt64(&num)

WaitGroup

Waits for goroutines to finish.

func main() {
   var wg sync.WaitGroup

   wg.Add(1) // Increases internal counter (+1 go routine)

   go worker(&wg)

   wg.Wait() // Will block code execution here, unless counter is set to 0

   println("Done!")
}

func worker(wg *sync.WaitGroup) {
   wg.Done() // Decrement counter by 1
}

Mutex

Mutex - mutual exclusion lock. Can be created as a part of a struct.

type Account struct { // embed Mutex object to allow to use its Public methods on Account
   sync.Mutex
   balance int
}

func main() {

   a := Account{balance: 0}

   var wg sync.WaitGroup

   wg.Add(100)

   for i := 0; i < 100; i++ {
      go worker(&a, &wg)
   }

   wg.Wait()

   println(a.balance)  // w/o mutex - balance can be 98, 99, 100 (data race happens)
}

func worker(account *Account, wg *sync.WaitGroup) {
   account.Lock()
   defer account.Unlock()  // convenient if method is long (but w/o defer - faster)

   account.balance += 1

   wg.Done()
}

RWMutex

Non-blocking read operations on resource.

Many can read data, while no one can edit it. While anyone edits data -
no one can read it.

a.RLock()  // lock resource for read
a.RLock()  // lock resource for read - works
a.Lock()   // lock resource for edit - doesn't work
a.RUnlock()// release resource
a.Lock()   // lock resource for edit - works

Map

Concurrent map with constant-time load, save, delete operations. Doesn't
require Mutex. Must not be copied - use pointer.

func main() {
    var m sync.Map
    
    for i := 0; i < 3; i++ {
        go func(i int) {
            for j := 0; ; j++ {
                m.Store(i, j)
            }
        }(i)
    }
    
    for i := 0; i < 10; i++ {
        m.Range(func(key, value interface{}) bool {
            fmt.Printf("%d: %d\t", key, value)
            return true
        })
        fmt.Println()
        time.Sleep(time.Second)
    }
}

time

start := time.Now()
/* ... */
secs := time.Since(start).Seconds()
func WaitForServer(url string) error {
    const timeout = 1 * time.Minute
    deadline := time.Now().Add(timeout)
    for tries := 0; time.Now().Before(deadline); tries++ {
        _, err := http.Head(url)
        if err == nil {
            return nil // success
        }
        log.Printf("server not responding (%s); retrying...", err)
        time.Sleep(time.Second << uint(tries)) // exponential back-off
    }
    return fmt.Errorf("server %s failed to respond after &s", url, timeout)
}

runtime

runtime.GOMAXPROCS(runtime.NumCPU()) // activate all CPU cores

runtime.Gosched()  // Allow runtime to pass control to other routine

Gosched() is important in concurrent programs while working in big
loops

flag

// Args returns the non-flag command-line arguments.
func Args() []string {}

Bool defines a bool flag with specified name, default value, and usage string.
func Bool(name string, value bool, usage string) *bool
// Bool creates new flag variable
var v = flag.Bool("v", false, "enable verbose mode")
var sep = flag.String("s", " ", "separator")

func main() {
    flag.Parse() // update the flag variables from their default values
    if *v {
        fmt.Println("verbose mode enabled")
    }
    fmt.Print(strings.Join(flag.Args(), *sep))
}

flag.Duration() creates a flag variable of type time.Duration and
allows the user to specify the duration in a variety of user-friendly
formats: ./cmd -flag 2m30s.

os

os.Args // []string containing CL arguments starting from CMD name itself
os.Getwd // Working directory

// Open returns *File that is used in subsequent reads by the bufio.NewScanner(*os.File)
func Open(name string) (*File, error) {}

// RemoveAll removes path and any children it contains.
func RemoveAll(path string) error {}

Failure to check the result of the close operation could cause serious
data loss to go unnoticed.

f, err := os.Create("file.txt")
if err != nil {
    return err
}

io.Copy(f, buf)

// Close file
err := f.Close()
if err != nil {
    return err
}

Network

TCP echo server (Async)

func main() {
    listener, err := net.Listen("tcp", ":3000") // Bind (connect + listen on OS Port)
    if err != nil {
      fmt.Println("Problem w/ creating server")
    }

    for {
      conn, err := listener.Accept()
      if err != nil {
         fmt.Println("Problem w/ connection")
         conn.Close()
         continue
      }

      fmt.Println("Connected")

      bufReader := bufio.NewReader(conn)

      go func() {
         defer conn.Close()

         for {
            // Reading from socket by bytes
            rbyte, err := bufReader.ReadByte()
            if err != nil {
               fmt.Println("Problem w/ reading from socket")
               break
            }

            fmt.Println(string(rbyte))
            conn.Write([]byte{rbyte})
         }
      }()
    }
}

HTTP Server

func main() {
    // Map request to a handler function
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        path := r.URL.Path

        c := &http.Client{} // create HTTP Client w/ default transport

        // Make request
        response, err := c.Get("http://setgetgo.com/get.php?Id=" + path[1:])
        if err != nil {
            log.Println(err)
        }

        defer response.Body.Close() // Connection is opened => Close Body after read, otherwise leaks may appear

        body, _ := ioutil.ReadAll(response.Body) // Read until EOF, return []byte

        w.WriteHeader(http.StatusOK)
        w.Write(body)
    })

    http.ListenAndServe(":3000", nil)
}

Server port can be set from CLI or from env.

JSON API Server

type apiRequest struct {
    Name string `json:"name"`
}

type apiResponse struct {
    Greeting string `json:"greeting"`
}

func main() {
    http.HandleFunc("/json", jsonHandler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

func jsonHandler(w http.ResponseWriter, r *http.Request) {

    var request apiRequest

    decoder := json.NewDecoder(r.Body)
    err := decoder.Decode(&request)
    if err != nil {
        log.Fatal(err)
    }

    w.Header().Set("Content-Type", "application/json")

    encoder := json.NewEncoder(w)
    err = encoder.Encode(apiResponse{"Hello " + request.Name})
    if err != nil {
        log.Fatal(err)
    }
}

Templates

text/template

Template - string or file containing one or more portions enclosed
in double braces {{...}} called actions.

  • . - current value
  • {{range .Items}}... {{end}} - loop
  • .Name | func - pipe field as parameter to a function

Producing output with a template:

  1. Parse the template into internal representation (done only once)
  2. Execute it on specific inputs

package main

import (
    "text/template"
    "time"
    "os"
    "log"
)

const templ = `Car List ({{len . }}):
{{range .}}----------------------------------------
Model:		{{.Model | printf "%.30s"}}
Age:		{{.Year | countAge}} years
Condition: 	{{.Condition}}
Owner:		{{.Owner.Name}} ({{.Owner.Phone}})
{{end}}`

type Owner struct {
    Name  string
    Phone string
}

type Car struct {
    Model     string
    Year      int
    Condition string
    Owner     *Owner
}

func countAge(year int) int {
    return time.Now().Year() - year
}

// template.Must accepts a template and an error, checks that the error is nil and returns a template
var report = template.Must(template.New("carlist").
    Funcs(template.FuncMap{"countAge": countAge}).
    Parse(templ))

func main() {
    cars := []Car{
        {"Volvo", 2000, "Used", &Owner{"Jack", "0123456"}},
        {"Lexus", 2010, "New", &Owner{"Helen", "0123456"}},
    }

    // cars will be `.` in template
    if err := report.Execute(os.Stdout, cars); err != nil {
        log.Fatal(err)
    }
}
Car List (2):
----------------------------------------
Model:          Volvo
Age:            17 years
Condition:      Used
Owner:          Jack (0123456)
----------------------------------------
Model:          Lexus
Age:            7 years
Condition:      New
Owner:          Helen (0123456)

html/template

Uses the same API as text/template but adds features for automatic
escaping of strings for HTML, JS, CSS or URL.

Inline

func main() {
    t1 := template.New("my-template") // Allocate new HTML template
    t1, _ = t1.Parse(`<div>{{.}}</div>`)
    t1.Execute(os.Stdout, "Inserted text")

    t2 := template.New("condition")
    t2, _ = t2.Parse(`{{if ne . "Hello"}}Not {{end}}Hello`)
    t2.Execute(os.Stdout, "Hello")
}

Separate file

Template in separate file template.html:

<html>
<head>
    <title>Shopping List</title>
</head>
<body>
<div class="root">
    {{if CheckAllInactive .}}
        <div>All items were bought.</div>
    {{else}}
        <div class="title">Items to buy:</div>
        {{range $index, $value := .}}
        <div class="item">
            <div class="item-name">{{.Name}}</div>
            <form action="/" method="post">
                <input type="hidden" name="id" value="{{$index}}"/>
                <input type="submit" {{if eq .Active false}}disabled{{end}} value="Bought!"/>
            </form>
        </div>
        {{end}}
    {{end}}
</div>
</body>
</html>
type Item struct {
    Name   string
    Active bool
}

func CheckAllInactive(items []Item) bool {
    for index := range items {
        if items[index].Active {
            return false
        }
    }
    return true
}

func main() {
    items := []Item{
        {"Bread", true},
        {"Eggs", true},
        {"Milk", true},
        {"Butter", true},
    }

    t1, err := template.New("template.html").Funcs(template.FuncMap{"CheckAllInactive": CheckAllInactive}).ParseFiles("template.html")
    if err != nil {
        log.Fatal("Can't expand template", err)
        return
    }

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        if r.Method == http.MethodPost {
            index, _ := strconv.ParseInt(r.FormValue("id"), 10, 0) // urlencoded -> string -> int
            items[index].Active = false
        }

        err = t1.Execute(w, items)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
        }
    })

    http.ListenAndServe(":3000", nil)
}

Cookies

Set-Cookie: session_id=ow42234e88vhg0f83hfs93r32' Expires=Mon, 4 Mar 2018 12:00:00 GMT

func main() {

    sessions := make(map[string]string)

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        sessionID, err := r.Cookie("session_id")
        if err == http.ErrNoCookie {
            w.Write([]byte(formTemplate))
            return
        } else if err != nil {
            log.Panic(err)
        }

        username, ok := sessions[sessionID.Value]
        if !ok {
            fmt.Fprint(w, "Session not found.")
        } else {
            fmt.Fprintf(w, "Welcome, "+username)
        }

    })

    http.HandleFunc("/get-cookie", func(w http.ResponseWriter, r *http.Request) {

        if r.Method == http.MethodPost {

            r.ParseForm()

            login := r.Form["login"][0]
            sessionId := strconv.Itoa(rand.Int())
            expiration := time.Now().Add(365 * 24 * time.Hour)
            cookie := http.Cookie{
                Name:    "session_id",
                Value:   sessionId,
                Expires: expiration,
            }

            sessions[sessionId] = login

            http.SetCookie(w, &cookie) // Cookie is set in headers, all content must be below
            http.Redirect(w, r, "/", http.StatusFound)

        }
    })

    http.ListenAndServe(":3000", nil)
}

DataBase

MySQL

MySQL Driver: import _ "github.com/go-sql-driver/myslq"

Connect

// Create structure of the DB
// No connections to the DB here
// Can use `interpolateParams=true` to change PREPARE+EXECUTE => QUERY
db, err := sql.Open("mysql", "root:password@tcp(localhost:3306)/testdb?charset=utf8&interpolateParams=true")
if err != nil {
    panic(err)
}

db.SetMaxOpenConns(10) // Optional; 10 connections can easily process 8000+ fast QPS

// Connect to the DB
if err = db.Ping(); err != nil {
    panic(err)
}

fmt.Println("Open connections to the DB:", db.Stats().OpenConnections)

Query() for SELECT many

// Query() returns multiple results
// Requires closing connection to DB
rows, err := db.Query("SELECT name, info FROM users")

// Iterate over rows
for rows.Next() {
    var name string
    var info sql.NullString // Now we can differentiate NULL and "" value returned from DB

    if err := rows.Scan(&name, &info); err != nil { // Populates `name` and `info` variables
        panic(err)
    }

    if info.Valid {
        fmt.Println("User:", name, info.String)
    } else {
        fmt.Println("User:", name, "No info") // `info` value is NULL
    }
}

rows.Close() // close connection, otherwise - leaks

QueryRow() for SELECT one

// QueryRow() returns single result
// Auto closes connection to DB
row := db.QueryRow("SELECT name, info FROM users WHERE id=?", 1)

var name string
var info string
if err := row.Scan(&name, &info); err != nil {
    panic(err)
}

fmt.Println("User:", name, info)

Exec() for INSERT/UPDATE

// Exec() executes query and returns num of changed rows
result, err := db.Exec("INSERT INTO users (name, info) VALUES (?, ?)", "Oleg Legun", "Some information")
if err != nil {
    panic(err)
}

affected, err := result.RowsAffected()
if err != nil {
    panic(err)
}

lastId, err := result.LastInsertId()
if err != nil {
    panic(err)
}

fmt.Println("Inserted values:", affected, "Last ID:", lastId)

Prepared statements: Prepare() + Exec()

// Prepared statements
// Send request to DB (PREPARE)
stmt, err := db.Prepare("UPDATE users SET info = ? WHERE id = ?")
if err != nil {
    panic(err)
}

// Now DB is waiting for values: info, id (EXECUTE)
result, err = stmt.Exec("Updated information", 5)
if err != nil {
    panic(err)
}

affected, err = result.RowsAffected()
if err != nil {
    panic(err)
}

fmt.Println("Updated values:", affected)

PostgreSQL

Postgres driver import _ "github.com/lib/pq"

Connect

// Create structure of the DB
// No connections to the DB here
db, err := sql.Open("postgres", "user=postgres password=password dbname=testdb sslmode=disable")
if err != nil {
    panic(err)
}

db.SetMaxOpenConns(10) // Optional; 10 connections can easily process 8000+ fast QPS

// Connect to the DB
if err = db.Ping(); err != nil {
    panic(err)
}

fmt.Println("Open connections to the DB:", db.Stats().OpenConnections)

Query() for SELECT many

// Query() returns multiple results
// Requires closing connection to DB
rows, err := db.Query("SELECT name, info FROM users")

// Iterate over rows
for rows.Next() {
    var name string
    var info sql.NullString // Now we can differentiate NULL and "" value returned from DB

    if err := rows.Scan(&name, &info); err != nil { // Populates `name` and `info` variables
        panic(err)
    }

    if info.Valid {
        fmt.Println("User:", name, info.String)
    } else {
        fmt.Println("User:", name, "No info") // `info` value is NULL
    }
}

rows.Close() // close connection, otherwise - leaks

QueryRow() for SELECT one

// QueryRow() returns single result
// Auto closes connection to DB
row := db.QueryRow("SELECT name, info FROM users WHERE id=$1", 1)

var name string
var info string
if err := row.Scan(&name, &info); err != nil {
    panic(err)
}

fmt.Println("User:", name, info)

QueryRow() for INSERT + LastInsertId()

// QueryRow() is used for INSERT queries, when we need to return lastId (pq doesn't support LastInsertId())
// Auto closes connection to DB
var lastId int64

err = db.QueryRow(
    "INSERT INTO users (name, info) VALUES ($1, $2) RETURNING id",
    "John",
    "Some Information",
).Scan(&lastId)

if err != nil {
    panic(err)
}

fmt.Println("Last insert Id:", lastId)

Exec() for INSERT/UPDATE

// Exec() executes query and returns num of changed rows
result, err := db.Exec(
    "UPDATE users SET name = $1, info = $2  WHERE id = $3",
    "Oleg Legun",
    "Updated information",
    4)
if err != nil {
    panic(err)
}

affected, err := result.RowsAffected()
if err != nil {
    panic(err)
}

fmt.Println("Updated values:", affected)

Prepared statements: Prepare() + Exec()

// Prepared statements
// Send request to DB (PREPARE)
stmt, err := db.Prepare("UPDATE users SET info = $1 WHERE id = $2")
if err != nil {
    panic(err)
}

// Now DB is waiting for values: info, id (EXECUTE)
result, err = stmt.Exec("Updated information", 5)
if err != nil {
    panic(err)
}

affected, err = result.RowsAffected()
if err != nil {
    panic(err)
}

fmt.Println("Updated values:", affected)

Memcached

Very efficient noSQL DB.

import "github.com/bradfitz/gomemcache/memcache"

var (
    memcacheClient *memcache.Client // Make it global - useful for functions
)

func getRecord(mkey string) *memcache.Item {
    println("get", mkey)

    item, err := memcacheClient.Get(mkey)
    if err == memcache.ErrCacheMiss {
        fmt.Println("Record not found.")
        return nil
    } else if err != nil {
        panic(err)
    }

    return item
}

func main() {
    MemcachedAddresses := []string{"127.0.0.1:11211"}
    memcacheClient = memcache.New(MemcachedAddresses...)

    // Get record
    mkey := "user_123"

    item := getRecord(mkey)
    fmt.Printf("Item:%+v\n", item)

    // Add record
    ttl := 5 // Seconds
    //ttl := 86400 // very useful for User sessions 24H

    // Set() - add record anyway
    err := memcacheClient.Set(&memcache.Item{
        Key:        mkey,
        Value:      []byte("Oleg"),
        Expiration: int32(ttl),
    })

    if err != nil {
        panic(err)
    }

    // Add() - add record if not exist
    err = memcacheClient.Add(&memcache.Item{
        Key:        "counter",
        Value:      []byte("1"),
        Expiration: int32(ttl),
    })

    if err == memcache.ErrNotStored {
        fmt.Println("Record not stored.") // Record was added before
    } else if err != nil {
        panic(err)
    }

    // Increment()
    newValue, err := memcacheClient.Increment("counter", 10)
    fmt.Println(newValue)
    newValue, err = memcacheClient.Decrement("counter", 5)
    fmt.Println(newValue)

    // GetMulti(): Get multiple records
    mkeys := []string{mkey, "counter"}
    items, err := memcacheClient.GetMulti(mkeys)
    if err != nil {
        panic(err)
    }
    fmt.Println("Map with multiple items:", items)
}

Redis

Heroku recommends import "github.com/garyburd/redigo/redis"

Rate limiter pattern

Cache methodology:

  1. Check for key in Redis
  2. If key does not exist
    • Rebuild cache: Make SQL query and save result to Redis (ttl n sec)


var (
    c redis.Conn
)

func getRecord(mkey string) string {
    fmt.Println("get", mkey)
    // Parse result (command reply) to a string
    item, err := redis.String(c.Do("GET", mkey))
    if err == redis.ErrNil {
        fmt.Println("Record not found (nil).")
        return ""
    } else if err != nil {
        panic(err)
    }

    return item
}

func main() {
    var err error

    // Connect
    c, err = redis.DialURL("redis://user:@localhost:6379/0")
    if err != nil {
        panic(err)
    }

    defer c.Close()

    // Get Record
    mkey := "name"

    item := getRecord(mkey)
    fmt.Println("Record:", item)

    // Add Record
    ttl := 60

    result, err := redis.String(c.Do("SET", "counter", 1, "EX", ttl))
    if err != nil {
        panic(err)
    }
    if result != "OK" {
        panic("result not OK: " + result)
    }

    // Increment (INCR/INCRBY/DECRBY)
    // If counter key is not set - Redis will create it with value: 0
    counter, err := redis.Int(c.Do("INCR", "counter"))
    if err != nil {
        panic(err)
    }
    fmt.Println("Counter:", counter)

    // MGET - Get multiple records
    keys := []interface{}{mkey, "counter", "nonexisting"}
    reply, err := redis.Strings(c.Do("MGET", keys...))
    if err != nil {
        panic(err)
    }
    fmt.Println(reply) // [Oleg Legun 1 nil]
}

Cache Invalidation

If cache is expired (key doesn't exist), and several processes are going
to rebuild it, we need to use locks, otherwise - DB will be overloaded!

Set new record like mkey+"lock" while rebuilding cache.

// If key was not found, lock it and try to rebuild it
result, err := redis.String(c.Do("SET", mkey+"_lock", name, "PX", 50, "NX"))
if result != "OK" {
    fmt.Println("Other user is rebuilding it!")
}

// Rebuild cache

// Delete _lock if operation was quicker than expected
n, err := redis.Int(c.Do("DEL", mkey+"lock"))

MongoDB

MongoDB driver: import "gopkg.in/mgo.v2"

import (
    "fmt"
    "gopkg.in/mgo.v2"
    "gopkg.in/mgo.v2/bson"
)

// MongoDB can have objects with any structure inside.
// In Go we need to explicitly describe each structure to successfully deserialize it.
// json is used when we serialize to JSON, bson - to Binary JSON (MongoDB type)
// Any extra keys that are present in MongoDB, but not in the `user` struct will be omitted
type user struct {
    ID   bson.ObjectId `json:"id" bson:"_id"` // _id - primary key (_) in MongoDB
    Name string        `json:"name" bson:"name"`
    Info string        `json:"info" bson:"info"`
}

var (
    sess *mgo.Session
)

func main() {
    var err error
    sess, err = mgo.Dial("mongodb://localhost")
    if err != nil {
        panic(err)
    }

    // Collection is like a table in MySQL
    collection := sess.DB("testdb").C("users")

    // Create index
    index := mgo.Index{
        Key: []string{"name"},
    }

    // Save index
    err = collection.EnsureIndex(index)
    if err != nil {
        panic(err)
    }

    // Add document
    if n, _ := collection.Count(); n == 0 {
        firstUser := &user{bson.NewObjectId(), "Oleg", "New information"}
        err = collection.Insert(firstUser)
        if err != nil {
            panic(err)
        }
    }

    // Get values
    var allUsers []user
    // Find all documents in collection (bson.M{empty} == ALL) and save them to allUsers
    err = collection.Find(bson.M{}).All(&allUsers)
    if err != nil {
        panic(err)
    }
    for i, v := range allUsers {
        fmt.Printf("user[%d]: %+v\n", i, v)
    }

    id := bson.NewObjectId()
    // bson.M{"_id": id} - set find condition
    var notExistentUser user

    // Find() returns *Query, .One() will Unmarshall values to &newUser
    err = collection.Find(bson.M{"_id": id}).One(&notExistentUser)
    if err != nil {
        //panic(err)
    }

    // Insert new document
    user1 := &user{id, "Jack", "Incorrect Information"}
    err = collection.Insert(user1)
    if err != nil {
        panic(err)
    }

    // Edit retrieved document
    user1.Info = "Correct Information"
    collection.Update(bson.M{"_id": user1.ID}, &user1)

    // Edit documents inside MongoDB
    collection.UpdateAll(
        bson.M{"name": "Jack"}, // Filter condition WHERE (for all Jacks) or  bson.M{"_id": id} - 1 document
        bson.M{
            "$set": bson.M{"info": "Correct Information"},
        },
    )
}

ORM

Not as effective as the traditional SQL queries. Can be slow for
high-loaded services.

ORM can generate suboptimal queries: we need 1 field, orm pulls all
fields.

Complex queries with several JOIN - in ORM can be very slow or even
not possible.

Values from SQL tables are automatically mapped to Go structures.

ORM is excellent for UPDATE'ing single user record (not for SELECT!).

import (
    "fmt"
    "github.com/jinzhu/gorm"
    _ "github.com/jinzhu/gorm/dialects/mysql"
)

var (
    db *gorm.DB
)

type User struct {
    ID   uint `sql:"AUTO_INCREMENT" gorm:"primary_key"`
    Name string
    Info string
}

func (u *User) TableName() string { // For GORM
    return "users"
}

func (u *User) BeforeSave() (err error) { // For GORM
    fmt.Println("Trigger on before save")
    return
}

func PrintByID(id uint) {
    u := User{}
    err := db.Find(&u, id).Error // SELECT * FROM users WHERE id = ?
    if err == gorm.ErrRecordNotFound {
        fmt.Println("Record no found", id)
    } else if err != nil {
        panic(err)
    }
    fmt.Printf("PrintByID: %v, data: %+v\n", id, u)
}

func main() {
    var err error

    // Connect
    db, err = gorm.Open("mysql", "root:password@tcp(localhost:20000)/testdb?charset=utf8")
    if err != nil {
        panic(err)
    }

    defer db.Close()

    db.DB() // ORM requires explicit connection to DB
    db.DB().Ping()

    PrintByID(1)
    PrintByID(100)

    // Select all records
    all := []User{} // SELECT * FROM users
    db.Find(&all)
    for i, v := range all {
        fmt.Printf("User %d: %+v\n", i, v)
    }

    // Create new record
    newUser := User{
        Name: "Jack",
        Info: "Some information",
    }

    db.NewRecord(newUser) // => returns `true` as primary key is blank
    db.Create(&newUser)

    PrintByID(newUser.ID)

    // Update record
    newUser.Info = "New information"
    db.Save(&newUser)
    PrintByID(newUser.ID)

}

Tests

Unit tests - most valuable tests.

CommandDescription
go testexecute *_test.go files
go test ./handlersTest package handlers
go test -vverbose mode (prints t.Log())
go test -run TestDivideexecute only *TestDivide* test case (supports RegExp)
go test -coverShow coverage percentage
go test -coverprofile=coverage.outExport coverage report to cover.out
go tool cover -func=coverage.outCoverage by functions based on cover.out
go tool cover -html=coverage.outOpen in Browser
go tool cover -html=coverage.out -o coverage.htmlConvert .out -> .html

Simple

package main

import "github.com/pkg/errors"

type User struct {
    Name    string
    Surname string
}

func Sum(a int, b int) int {
    return a + b
}

func AddUser(usersPtr *[]User, name string, surname string) {
    *usersPtr = append(*usersPtr, User{name, surname})
}

func Divide(a int, b int) (int, error) {
    if b == 0 {
        return 0, errors.New("Zero division error.")
    }
    return a / b, nil
}
package main

import (
    "github.com/pkg/errors"
    "reflect"
    "testing"
)

func TestSum(t *testing.T) {
    if Sum(1, 2) != 3 {
        // Errorf() generates an error and continues to execute current test case
        t.Errorf("Expected %d, got %d", 3, Sum(1, 2))
    }
}

func TestAddUser(t *testing.T) {
    users := []User{}

    AddUser(&users, "Oleg", "Legun")

    if len(users) != 1 {
        // Fatalf() generates an error and returns from current function
        t.Fatalf("Expected len = %d, got %d", 1, len(users))
    }

    expected := []User{
        {
            Name:    "Oleg",
            Surname: "Legun",
        },
    }

    if !reflect.DeepEqual(users, expected) {
        t.Errorf("Expected %+v, got %+v", expected, users)
    }
}

// Table testing
func TestTable(t *testing.T) {
    t.Skip()
    testCases := []struct {
        A      int
        B      int
        Result int
        Err    error
    }{
        {10, 5, 2, nil},
        {-10, 2, -5, nil},
        {10, 0, 0, errors.New("Zero division error.")},
    }

    for _, testCase := range testCases {
        res, err := Divide(testCase.A, testCase.B)
        if res != testCase.Result {
            t.Errorf("Divide(%d, %d) => %d, want %d", testCase.A, testCase.B, res, testCase.Result)
        }

        if err != nil && err.Error() != testCase.Err.Error() {
            t.Errorf("Divide(%d, %d) => Error: %v, want Error: %v", testCase.A, testCase.B, err.Error(), testCase.Err.Error())
        }
    }
}
  • t.Parallel() - makes tests execution concurrent
  • t.Log("Message") displays message in verbose mode -v.
  • t.Skip() skips current test.


Test HTTP handlers

To make code testable:

  • break the code into pure functions
  • keep all external data in separate files, that can be mocked

PROJECT
   ├── main/
   │    └── main.go
   └── handlers/
         ├── adduser.go
         ├── adduser_test.go
         ├── handlers.go
         └── index.go
// main.go
package main

import (
    "../handlers"
    "net/http"
)

func main() {
    handler := handlers.Handler{
        Users: &[]handlers.User{},
    }

    http.HandleFunc("/", handler.HandleIndex)
    http.HandleFunc("/adduser", handler.HandleAddUser)
    http.ListenAndServe(":3000", nil)
}
// handlers/handlers.go
package handlers

type User struct {
    Name    string `json:"name"`
    Surname string `json:"surname"`
    Age     uint8  `json:"age"`
}

type Handler struct {
    Users *[]User
}
// handlers/index.go
package handlers

import (
	"encoding/json"
	"log"
	"net/http"
)

func (h Handler) HandleIndex(w http.ResponseWriter, r *http.Request) {
    switch r.Method {
    case http.MethodGet:
        json, err := json.Marshal(*h.Users)
        if err != nil {
            log.Panic("JSON marshalling error")
        }

        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(json))

    case http.MethodPost:
        w.WriteHeader(http.StatusMethodNotAllowed)
    }
}
// handlers/adduser.go
package handlers

import (
    "log"
    "net/http"
    "strconv"
)

func (h Handler) HandleAddUser(w http.ResponseWriter, r *http.Request) {
    switch r.Method {
    case http.MethodGet:
        q := r.URL.Query()

        name := q["name"][0]
        surname := q["surname"][0]

        age, err := strconv.Atoi(q["age"][0])
        if err != nil {
            log.Panic("Invalid age.")
        }

        *h.Users = append(*h.Users, User{name, surname, uint8(age)})

    case http.MethodPost:
        w.Write([]byte("Not implemented yet."))

    default:
        w.WriteHeader(http.StatusMethodNotAllowed)
    }
}
// handlers/adduser_test.go
package handlers

import (
    "net/http"
    "net/http/httptest"
    "strings"
    "testing"
)

func TestHandler_HandleAddUser(t *testing.T) {
    // Create handler w/ test data
    h := Handler{
        Users: &[]User{},
    }

    // Create handler from a function
    handler := http.HandlerFunc(h.HandleAddUser)

    r, err := http.NewRequest("GET", "/adduser?name=Oleg&surname=Legun&age=26", strings.NewReader(""))
    if err != nil {
        t.Errorf("Error %v", err)
    }

    // recorder keeps all http requests
    w := httptest.NewRecorder()

    handler.ServeHTTP(w, r)

    if w.Code != http.StatusOK {
        t.Fatalf("Expected %d, got %d", http.StatusOK, w.Code)
    }
}

func TestHandler_HandleIndex(t *testing.T) {
    // Create handler w/ test data
    handler := Handler{
        Users: &[]User{
            {"Oleg", "Legun", 26},
        },
    }

    // Create handler from a function
    h := http.HandlerFunc(handler.HandleIndex)

    r, err := http.NewRequest("GET", "/", strings.NewReader(""))
    if err != nil {
        t.Errorf("Error %v", err)
    }

    // recorder keeps all http requests
    w := httptest.NewRecorder()

    h.ServeHTTP(w, r)

    if w.Code != http.StatusOK {
        t.Fatalf("Expected %d, got %d", http.StatusOK, w.Code)
    }

    expected := `[{"name":"Oleg","surname":"Legun","age":26}]`

    // Can also use DeepEqual() instead of a string
    if w.Body.String() != expected {
        t.Errorf("Expected %s, got %s", expected, w.Body.String())
    }
}

Mocks

https://github.com/golang/mock

We can test how our program works w/ external dependency by using a
generated mock based on its interface.

Package must be available on GOPATH.

CommandDescription
mockgen -destination=./mocks/users_mock.go users UserInterfaceGenerate mocks based on UserInterface from users package
// External Dependency `/users` on GOPATH
package users

type User struct {
    ID   int
    Name string
}

func (u User) GetName() string {
    return u.Name
}

func (u *User) SetName(name string) {
    u.Name = name
}

type UserInterface interface {
    GetName() string
    SetName(string)
}
// users_test.go
package users

import (
    "./mocks"
    "./users"
    "fmt"
    "github.com/golang/mock/gomock"
    "testing"
)

func SetAndGetName(u users.UserInterface) {
    u.SetName("Oleg")  // 1st operation
    name := u.GetName()
    fmt.Println(name)  // 2nd operation
}

func TestSetAndGetName(t *testing.T) {
    mockCtrl := gomock.NewController(t)  // pass `t` to enable logging to the console if gomock fails
    defer mockCtrl.Finish()  // Check all methods calls after SetAndGetName() finishes and log PASS/FAIL

    testUser := mocks.NewMockUserInterface(mockCtrl)

    // Record a particular sequence of methods calls, that must be executed
    testUser.EXPECT().SetName("Oleg") // record 1st operation
    testUser.EXPECT().GetName().Return("Oleg")  // record 2nd operation

    SetAndGetName(testUser) // Pass `testUser` mock to the function to test it
}
`