连接google cloud vm其他主机类似

1.ssh-keygen -C '邮箱' -f ./aaa
会在当前路径下生产aaa和aaa.pub
2.执行ssh-keygen会在.ssh生成id_rsa和id_rsa.pub
3.在.ssh下面创建文件authorized_keys，讲上面aaa中的文件贴进去
4.打开xshell在用户身份验证中，选择public key，在用户密钥中选择本地文件aaa（跟上面的aaa一样）
5.即可直接进行连接，winscp类似
6.需要编辑google cloud vm，在ssh中加上aaa中的私钥

shadowsocks 搭建

参见
https://xfabs.github.io/2016/08/02/shadowsocks/


主要apt-get install python-pip
pip install shadowsocks


{
    "server":"0.0.0.0",
    "server_port":1111,
    "local_address": "127.0.0.1",
    "local_port：":1080,
    "password":"mima",
    "timeout":300,
    "method":"aes-256-cfb",
    "fast_open": false
}


ssserver -c /etc/ss.json -d start
ssserver -c /etc/ss.json -d stop

https://blog.jamespan.me/posts/enable-tcp-bbr-on-google-cloud

一定要开启上面的bbr

sudo bash -c 'echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf'
sudo bash -c 'echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf'
sudo sysctl -p

spark-sehll client模式

spark-shell --master yarn --deploy-mode client

快速排序golang

package main

import "fmt"

func qsort(data []int) {
if len(data) <= 1 {
return
}
mid, i := data[0], 1
head, tail := 0, len(data)-1
for i = 1; i <= tail; {
                if data[i] > mid {
data[i], data[tail] = data[tail], data[i]
tail--
} else {
data[i], data[head] = data[head], data[i]
head++
i++
}
                }
data[head] = mid
qsort(data[:head])
qsort(data[head+1:])
}
func main() {
fmt.Println("Hello, 世界")

a:=[]int{1,2,3}
qsort(a)
for x:=range(a){
fmt.Println(x)}
}

线上spark和sqoop问题修复

线上sqoop日志错误如下，https://stackoverflow.com/questions/38649071/getting-an-file-exists-error-while-import-into-hive-using-sqoop
https://stackoverflow.com/questions/38649071/getting-an-file-exists-error-while-import-into-hive-using-sqoop

参见这两个，大概意思是说sqoop导数据的时候，会将数据先放到一个临时路径下，然后在挪过去。如果这个临时路径存在，就会报错，哪怕选线是overwrite也不行；

spark错误

http://blog.csdn.net/a921122/article/details/45095845
https://stackoverflow.com/questions/32820087/spark-multiple-spark-submit-in-parallel

scala错误

java.lang.NoSuchMethodError: scala.runtime.IntRef.create(I)Lscala/runtime/IntRef

scala版本问题

sse操作

https://msdn.microsoft.com/en-us/library/ybhzf6dk(v=vs.71).aspx
https://msdn.microsoft.com/en-us/library/yd9wecaa(v=vs.120).aspx
https://msdn.microsoft.com/en-us/library/yd9wecaa(v=vs.120).aspx
https://msdn.microsoft.com/en-us/library/2x1se8ha(VS.80).aspx
https://msdn.microsoft.com/en-us/library/22kbk6t9(VS.80).aspx
https://msdn.microsoft.com/en-us/library/c9848chc(v=vs.90).aspx
https://msdn.microsoft.com/en-us/library/yd9wecaa(v=vs.120).aspx
https://msdn.microsoft.com/en-us/library/ybhzf6dk(v=vs.71).aspx
https://msdn.microsoft.com/en-us/library/tk1t2tbz(v=vs.80).aspx