PostgreSQL中Slony-I同步复制部署教程

前言

本文主要介绍了关于PostgreSQL中Slony-I同步复制部署的相关内容,分享出来供大家参考学习,下面话不多说了,来一起看看详细的介绍吧

本次测试环境

IP 10.189.102.118 10.189.100.195 10.189.100.226
PGHOME /usr/local/pgsql /usr/local/pgsql /usr/local/pgsql
Role Origin,Providers Subscribers,Providers Subscribers
Database region_il  region_il  region_il 
Table
il_longlive.entity

il_longlive.entity_result

il_longlive.result

il_longlive.result_logbook


il_longlive.entity

il_longlive.entity_result

il_longlive.result

il_longlive.result_logbook


il_longlive.entity

il_longlive.entity_result

il_longlive.result

il_longlive.result_logbook

Sequences
il_longlive.entity_entity_id_seq

il_longlive.entity_result_id_seq

il_longlive.result_id_seq

il_longlive.result_logbook_record_id_seq


il_longlive.entity_entity_id_seq

il_longlive.entity_result_id_seq

il_longlive.result_id_seq

il_longlive.result_logbook_record_id_seq

il_longlive.entity_entity_id_seq

il_longlive.entity_result_id_seq

il_longlive.result_id_seq

il_longlive.result_logbook_record_id_seq

Slony-I一般要求表有主键,或者唯一键,Slony-I会自动识别主键,如果表没有主键,需要手工指定唯一键,如果没有指定唯一键,则不能复制,下图为Slony-I架构图.

1. 查看主库表信息

region_il=# \d
   List of relations
 Schema |  Name  | Type | Owner
-------------+------------------------------+----------+----------
 il_longlive | entity   | table | postgres
 il_longlive | entity_entity_id_seq  | sequence | postgres
 il_longlive | entity_result  | table | postgres
 il_longlive | entity_result_id_seq  | sequence | postgres
 il_longlive | result   | table | postgres
 il_longlive | result_id_seq  | sequence | postgres
 il_longlive | result_logbook  | table | postgres
 il_longlive | result_logbook_record_id_seq | sequence | postgres
 il_longlive | result_vendor_schema  | table | postgres
 il_longlive | result_vendor_schema_id_seq | sequence | postgres
 il_longlive | snapshot   | table | postgres
 il_longlive | snapshot_id_seq  | sequence | postgres
(12 rows)

region_il=# \d entity
     Table "il_longlive.entity"
 Column |  Type  |    Modifiers
--------------+------------------------+------------------------------------------------------------
 entity_id | integer  | not null default nextval('entity_entity_id_seq'::regclass)
 country_code | character varying(3) |
 state_code | character varying(50) |
 name  | character varying(128) |
 islive | boolean  |
Indexes:
 "entity_pkey" PRIMARY KEY, btree (entity_id)

region_il=# select count(*) from entity;
 count
---------
(1 row)

2. 在所有节点安装Slony-I

$ wget http://www.slony.info/downloads/2.2/source/slony1-2.2.4.tar.bz2
$ tar -jxf slony1-2.2.4.tar.bz2
$ cd slony1-2.2.4
$ ./configure --prefix=/usr/local/pgsql --with-pgconfigdir=/usr/local/pgsql/bin --with-perltools
$ make
$ make install

3. 在所有节点创建Slony-I使用的专用用户

$ psql -c "CREATE USER slonyadmin WITH SUPERUSER REPLICATION LOGIN PASSWORD '888888';"

4. 在所有节点设置允许Slony-I用户远程登录

$ grep slonyadmin /usr/local/pgsql/data/pg_hba.conf
host all  slonyadmin  0.0.0.0/0  md5

5. 在所有节点的对应数据库创建plpgsql语言

$ createlang plpgsql -d region_il 

6. 在主节点创建Slony-I配置文件,并复制到所有其他节点

$ cat /usr/local/pgsql/etc/slon_tools.conf
if ($ENV{"SLONYNODES"}) {
 require $ENV{"SLONYNODES"};
} else {
 $CLUSTER_NAME = 'region_il';
 $PIDFILE_DIR = '/usr/local/pgsql/data';
 $LOGDIR = '/usr/local/pgsql/data/pg_log';
 $MASTERNODE = 1;
 $DEBUGLEVEL = 0;
	$START_WATCHDOG = 1;
 $SLEEP_TIME = 60;
 $WATCHDOG_VERBOSE = 0;

 add_node(node => 1,
  host => '10.189.102.118',
  dbname => 'region_il',
  port => 5432,
  user => 'slonyadmin',
  password => '888888',);

 add_node(node => 2,
  parent => 1,
  host => '10.189.100.195',
  dbname => 'region_il',
  port => 5432,
  user => 'slonyadmin',
  password => '888888',);

 add_node(node => 3,
  parent => 2,
  host => '10.189.100.226',
  dbname => 'region_il',
  port => 5432,
  user => 'slonyadmin',
  password => '888888',);
}

$SLONY_SETS = {
 "il_longlive_schema" => {
 "set_id" => 1,
 "table_id" => 101,
 "sequence_id" => 101,
 "pkeyedtables" => [
    'il_longlive.entity',
    'il_longlive.entity_result',
    'il_longlive.result',
    'il_longlive.result_logbook',
    ],

 "keyedtables" => {},

 "sequences" => ['il_longlive.entity_entity_id_seq',
		 'il_longlive.entity_result_id_seq',
			'il_longlive.result_id_seq',
			'il_longlive.result_logbook_record_id_seq',
   ],
},

};

if ($ENV{"SLONYSET"}) {
 require $ENV{"SLONYSET"};
}

# Please do not add or change anything below this point.
 1;
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.195:/usr/local/pgsql/etc/
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.226:/usr/local/pgsql/etc/

7. 将需要复制的表结构从主节点导出并导入到所有辅节点中

$ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.195 -U slonyadmin region_il
$ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.226 -U slonyadmin region_il

 8. 在主节点初始化Slony-I集群

$ /usr/local/pgsql/bin/slonik_init_cluster 

# INIT CLUSTER
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 init cluster (id = 1, comment = 'Node 1 - region_il@10.189.102.118');

# STORE NODE
 store node (id = 2, event node = 1, comment = 'Node 2 - region_il@10.189.100.195');
 store node (id = 3, event node = 1, comment = 'Node 3 - region_il@10.189.100.226');
 echo 'Set up replication nodes';

# STORE PATH
 echo 'Next: configure paths for each node/origin';
 store path (server = 1, client = 2, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 1, client = 3, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 2, client = 1, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 2, client = 3, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 3, client = 1, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 3, client = 2, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 echo 'Replication nodes prepared';
 echo 'Please start a slon replication daemon for each node';
$ /usr/local/pgsql/bin/slonik_init_cluster | slonik
<stdin>:7: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:10: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:11: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:12: Set up replication nodes
<stdin>:15: Next: configure paths for each node/origin
<stdin>:22: Replication nodes prepared
<stdin>:23: Please start a slon replication daemon for each node

9. 在所有节点启动slon进程

在主节点启动slon进程

$ slon_start 1
Invoke slon for node 1 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node1/region_il-2017-06-14.log 2>&1 &
Slon successfully started for cluster region_il, node node1
PID [2454]
Start the watchdog process as well...
$ slon_status 1
Slon is running for the 'region_il' cluster on node1.
$ ps -ef | grep slon
postgres 2454 1 0 22:54 pts/1 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 2455 2454 0 22:54 pts/1 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 2459 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24667) idle
postgres 2467 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24669) idle
postgres 2468 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24673) idle
postgres 2469 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24675) idle
postgres 2470 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24677) idle
postgres 2471 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24681) idle
postgres 2473 1 0 22:54 pts/1 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node1 60

在辅节点1启动slave进程

$ slon_start 2
Invoke slon for node 2 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node2/region_il-2017-06-14.log 2>&1 &
Slon successfully started for cluster region_il, node node2
PID [15535]
Start the watchdog process as well...
$ slon_status 2
Slon is running for the 'region_il' cluster on node2.
$ ps -ef | grep slon
postgres 15515 12843 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(38804) idle
postgres 15535 1 0 22:57 pts/0 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 15536 15535 0 22:57 pts/0 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 15540 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54629) idle
postgres 15546 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54631) idle
postgres 15547 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54633) idle
postgres 15548 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54635) idle
postgres 15549 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54639) idle
postgres 15552 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54643) idle
postgres 15555 1 0 22:57 pts/0 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60
postgres 15559 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(38812) idle

在辅节点2启动slave进程

$ slon_start 3
Invoke slon for node 3 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-14.log 2>&1 &
Slon successfully started for cluster region_il, node node3
PID [19212]
Start the watchdog process as well...
$ slon_status 3
Slon is running for the 'region_il' cluster on node3.
$ ps -ef | grep slon
postgres 6033 1 0 Jun02 ? 00:00:02 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 6050 1 0 Jun02 ? 00:00:02 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60
postgres 19059 15237 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63615) idle
postgres 19109 15237 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(26781) idle
postgres 19212 1 0 23:00 pts/2 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 19213 19212 0 23:00 pts/2 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 19217 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41213) idle
postgres 19225 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41215) idle
postgres 19226 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41219) idle
postgres 19227 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41221) idle
postgres 19228 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41223) idle
postgres 19229 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41227) idle
postgres 19232 1 0 23:00 pts/2 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node3 60
postgres 19234 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63635) idle
postgres 19235 6033 0 23:00 ? 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 19239 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63637) idle

10. 在主节点创建复制集

$ /usr/local/pgsql/bin/slonik_create_set 1
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

# TABLE ADD KEY

# CREATE SET
 create set (id = 1, origin = 1, comment = 'Set 1 (il_longlive_schema) for region_il');

# SET ADD TABLE
 echo 'Subscription set 1 (il_longlive_schema) created';
 echo 'Adding tables to the subscription set';
 set add table (set id = 1, origin = 1, id = 101,
   full qualified name = 'il_longlive.entity',
   comment = 'Table il_longlive.entity with primary key');
 echo 'Add primary keyed table il_longlive.entity';
 set add table (set id = 1, origin = 1, id = 102,
   full qualified name = 'il_longlive.entity_result',
   comment = 'Table il_longlive.entity_result with primary key');
 echo 'Add primary keyed table il_longlive.entity_result';
 set add table (set id = 1, origin = 1, id = 103,
   full qualified name = 'il_longlive.result',
   comment = 'Table il_longlive.result with primary key');
 echo 'Add primary keyed table il_longlive.result';
 set add table (set id = 1, origin = 1, id = 104,
   full qualified name = 'il_longlive.result_logbook',
   comment = 'Table il_longlive.result_logbook with primary key');
 echo 'Add primary keyed table il_longlive.result_logbook';

# SET ADD SEQUENCE
 echo 'Adding sequences to the subscription set';
 set add sequence (set id = 1, origin = 1, id = 101,
   full qualified name = 'il_longlive.entity_entity_id_seq',
   comment = 'Sequence il_longlive.entity_entity_id_seq');
 echo 'Add sequence il_longlive.entity_entity_id_seq';
 set add sequence (set id = 1, origin = 1, id = 102,
   full qualified name = 'il_longlive.entity_result_id_seq',
   comment = 'Sequence il_longlive.entity_result_id_seq');
 echo 'Add sequence il_longlive.entity_result_id_seq';
 set add sequence (set id = 1, origin = 1, id = 103,
   full qualified name = 'il_longlive.result_id_seq',
   comment = 'Sequence il_longlive.result_id_seq');
 echo 'Add sequence il_longlive.result_id_seq';
 set add sequence (set id = 1, origin = 1, id = 104,
   full qualified name = 'il_longlive.result_logbook_record_id_seq',
   comment = 'Sequence il_longlive.result_logbook_record_id_seq');
 echo 'Add sequence il_longlive.result_logbook_record_id_seq';
 echo 'All tables added';
$ /usr/local/pgsql/bin/slonik_create_set 1 | slonik
<stdin>:12: Subscription set 1 (il_longlive_schema) created
<stdin>:13: Adding tables to the subscription set
<stdin>:17: Add primary keyed table il_longlive.entity
<stdin>:21: Add primary keyed table il_longlive.entity_result
<stdin>:25: Add primary keyed table il_longlive.result
<stdin>:29: Add primary keyed table il_longlive.result_logbook
<stdin>:32: Adding sequences to the subscription set
<stdin>:36: Add sequence il_longlive.entity_entity_id_seq
<stdin>:40: Add sequence il_longlive.entity_result_id_seq
<stdin>:44: Add sequence il_longlive.result_id_seq
<stdin>:48: Add sequence il_longlive.result_logbook_record_id_seq
<stdin>:49: All tables added

11. 在主节点创建复制订阅者

$ slonik_subscribe_set 1 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

 subscribe set (id = 1, provider = 1, receiver = 2, forward = yes);
 echo 'Subscribed nodes to set 1';
$ slonik_subscribe_set 1 2 | slonik
<stdin>:7: Subscribed nodes to set 1
$ slonik_subscribe_set 1 3
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

 subscribe set (id = 1, provider = 2, receiver = 3, forward = yes);
 echo 'Subscribed nodes to set 1';
$ slonik_subscribe_set 1 3 | slonik
<stdin>:7: Subscribed nodes to set 1

12.查看各节点Slony-I日志信息

主节点查看Slony-I日志信息

$ cat region_il-2017-06-14.log
2017-06-14 22:54:42 PDT CONFIG main: slon version 2.2.4 starting up
2017-06-14 22:54:42 PDT INFO slon: watchdog process started
2017-06-14 22:54:42 PDT CONFIG slon: watchdog ready - pid = 2454
2017-06-14 22:54:42 PDT CONFIG slon: worker process created - pid = 2455
2017-06-14 22:54:42 PDT CONFIG main: Integer option vac_frequency = 3
2017-06-14 22:54:42 PDT CONFIG main: Integer option log_level = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval = 1000
2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval_timeout = 10000
2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_group_maxsize = 20
2017-06-14 22:54:42 PDT CONFIG main: Integer option syslog = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option quit_sync_provider = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option remote_listen_timeout = 300
2017-06-14 22:54:42 PDT CONFIG main: Integer option monitor_interval = 500
2017-06-14 22:54:42 PDT CONFIG main: Integer option explain_interval = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_idle = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_interval = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_count = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option apply_cache_size = 100
2017-06-14 22:54:42 PDT CONFIG main: Boolean option log_pid = 0
2017-06-14 22:54:42 PDT CONFIG main: Boolean option log_timestamp = 1
2017-06-14 22:54:42 PDT CONFIG main: Boolean option tcp_keepalive = 1
2017-06-14 22:54:42 PDT CONFIG main: Boolean option monitor_threads = 1
2017-06-14 22:54:42 PDT CONFIG main: Real option real_placeholder = 0.000000
2017-06-14 22:54:42 PDT CONFIG main: String option cluster_name = region_il
2017-06-14 22:54:42 PDT CONFIG main: String option conn_info = host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888
2017-06-14 22:54:42 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node1.pid
2017-06-14 22:54:42 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2017-06-14 22:54:42 PDT CONFIG main: String option archive_dir = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option sql_on_connection = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option lag_interval = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option command_on_logarchive = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option syslog_facility = LOCAL0
2017-06-14 22:54:42 PDT CONFIG main: String option syslog_ident = slon
2017-06-14 22:54:42 PDT CONFIG main: String option cleanup_interval = 10 minutes
2017-06-14 22:54:42 PDT CONFIG main: local node id = 1
2017-06-14 22:54:42 PDT INFO main: main process started
2017-06-14 22:54:42 PDT CONFIG main: launching sched_start_mainloop
2017-06-14 22:54:42 PDT CONFIG main: loading current cluster configuration
2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=2 no_comment='Node 2 - region_il@10.189.100.195'
2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=3 no_comment='Node 3 - region_il@10.189.100.226'
2017-06-14 22:54:42 PDT CONFIG storePath: pa_server=2 pa_client=1 pa_conninfo="host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:54:42 PDT CONFIG storePath: pa_server=3 pa_client=1 pa_conninfo="host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG main: last local event sequence = 5000000007
2017-06-14 22:54:42 PDT CONFIG main: configuration complete - starting threads
2017-06-14 22:54:42 PDT INFO localListenThread: thread starts
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG enableNode: no_id=2
2017-06-14 22:54:42 PDT CONFIG enableNode: no_id=3
2017-06-14 22:54:42 PDT INFO main: running scheduler mainloop
2017-06-14 22:54:42 PDT INFO remoteWorkerThread_3: thread starts
2017-06-14 22:54:42 PDT INFO remoteListenThread_3: thread starts
2017-06-14 22:54:42 PDT CONFIG cleanupThread: thread starts
2017-06-14 22:54:42 PDT INFO syncThread: thread starts
2017-06-14 22:54:42 PDT INFO monitorThread: thread starts
2017-06-14 22:54:42 PDT INFO remoteListenThread_2: thread starts
2017-06-14 22:54:42 PDT INFO remoteWorkerThread_2: thread starts
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG cleanupThread: bias = 60
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:57:50 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:57:50 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:50 PDT INFO remoteWorkerThread_2: SYNC 5000000003 done in 0.014 seconds
2017-06-14 22:57:58 PDT INFO remoteWorkerThread_2: SYNC 5000000004 done in 0.003 seconds
2017-06-14 22:58:11 PDT INFO remoteWorkerThread_2: SYNC 5000000005 done in 0.002 seconds
2017-06-14 22:58:19 PDT INFO remoteWorkerThread_2: SYNC 5000000006 done in 0.002 seconds
2017-06-14 22:58:27 PDT INFO remoteWorkerThread_2: SYNC 5000000007 done in 0.003 seconds
2017-06-14 22:58:35 PDT INFO remoteWorkerThread_2: SYNC 5000000008 done in 0.002 seconds
2017-06-14 22:58:43 PDT INFO remoteWorkerThread_2: SYNC 5000000009 done in 0.002 seconds
2017-06-14 22:59:01 PDT INFO remoteWorkerThread_2: SYNC 5000000010 done in 0.003 seconds
2017-06-14 22:59:09 PDT INFO remoteWorkerThread_2: SYNC 5000000011 done in 0.002 seconds
2017-06-14 22:59:17 PDT INFO remoteWorkerThread_2: SYNC 5000000012 done in 0.002 seconds
2017-06-14 22:59:25 PDT INFO remoteWorkerThread_2: SYNC 5000000013 done in 0.002 seconds
2017-06-14 22:59:38 PDT INFO remoteWorkerThread_2: SYNC 5000000014 done in 0.003 seconds
2017-06-14 22:59:51 PDT INFO remoteWorkerThread_2: SYNC 5000000015 done in 0.002 seconds
2017-06-14 22:59:54 PDT INFO remoteWorkerThread_2: SYNC 5000000016 done in 0.002 seconds
2017-06-14 23:00:05 PDT INFO remoteWorkerThread_2: SYNC 5000000017 done in 0.002 seconds
2017-06-14 23:00:19 PDT INFO remoteWorkerThread_2: SYNC 5000000018 done in 0.003 seconds
2017-06-14 23:00:32 PDT INFO remoteWorkerThread_2: SYNC 5000000019 done in 0.003 seconds
2017-06-14 23:00:35 PDT INFO remoteWorkerThread_2: SYNC 5000000020 done in 0.003 seconds
2017-06-14 23:00:43 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:00:43 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:43 PDT INFO remoteWorkerThread_3: SYNC 5000000003 done in 0.045 seconds
2017-06-14 23:00:47 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:47 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:47 PDT INFO remoteWorkerThread_2: SYNC 5000000021 done in 0.014 seconds
2017-06-14 23:00:55 PDT INFO remoteWorkerThread_3: SYNC 5000000004 done in 0.002 seconds
2017-06-14 23:00:55 PDT INFO remoteWorkerThread_2: SYNC 5000000022 done in 0.005 seconds
2017-06-14 23:01:08 PDT INFO remoteWorkerThread_3: SYNC 5000000005 done in 0.002 seconds
2017-06-14 23:01:08 PDT INFO remoteWorkerThread_2: SYNC 5000000023 done in 0.005 seconds
2017-06-14 23:01:16 PDT INFO remoteWorkerThread_3: SYNC 5000000006 done in 0.002 seconds
2017-06-14 23:01:20 PDT INFO remoteWorkerThread_3: SYNC 5000000007 done in 0.002 seconds
2017-06-14 23:01:20 PDT INFO remoteWorkerThread_2: SYNC 5000000024 done in 0.005 seconds
2017-06-14 23:01:24 PDT INFO remoteWorkerThread_2: SYNC 5000000025 done in 0.005 seconds
2017-06-14 23:01:38 PDT INFO remoteWorkerThread_3: SYNC 5000000008 done in 0.003 seconds
2017-06-14 23:01:38 PDT INFO remoteWorkerThread_2: SYNC 5000000026 done in 0.005 seconds
2017-06-14 23:01:41 PDT INFO remoteWorkerThread_3: SYNC 5000000009 done in 0.002 seconds
2017-06-14 23:01:44 PDT INFO remoteWorkerThread_2: SYNC 5000000027 done in 0.004 seconds
2017-06-14 23:01:52 PDT INFO remoteWorkerThread_3: SYNC 5000000010 done in 0.003 seconds
2017-06-14 23:01:55 PDT INFO remoteWorkerThread_2: SYNC 5000000028 done in 0.005 seconds
2017-06-14 23:02:08 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:02:08 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:02:08 PDT INFO remoteWorkerThread_2: SYNC 5000000029 done in 0.012 seconds
2017-06-14 23:02:08 PDT INFO remoteWorkerThread_3: SYNC 5000000011 done in 0.016 seconds
2017-06-14 23:02:11 PDT INFO remoteWorkerThread_3: SYNC 5000000012 done in 0.004 seconds
2017-06-14 23:02:19 PDT INFO remoteWorkerThread_2: SYNC 5000000030 done in 0.005 seconds
2017-06-14 23:02:22 PDT INFO remoteWorkerThread_3: SYNC 5000000013 done in 0.005 seconds
2017-06-14 23:02:25 PDT INFO remoteWorkerThread_2: SYNC 5000000031 done in 0.004 seconds
2017-06-14 23:02:38 PDT INFO remoteWorkerThread_2: SYNC 5000000032 done in 0.004 seconds
2017-06-14 23:02:38 PDT INFO remoteWorkerThread_3: SYNC 5000000014 done in 0.005 seconds
2017-06-14 23:02:46 PDT INFO remoteWorkerThread_3: SYNC 5000000015 done in 0.005 seconds
2017-06-14 23:02:50 PDT INFO remoteWorkerThread_3: SYNC 5000000016 done in 0.004 seconds
2017-06-14 23:02:50 PDT INFO remoteWorkerThread_2: SYNC 5000000033 done in 0.005 seconds
2017-06-14 23:02:54 PDT INFO remoteWorkerThread_2: SYNC 5000000034 done in 0.004 seconds
2017-06-14 23:03:08 PDT INFO remoteWorkerThread_3: SYNC 5000000017 done in 0.004 seconds
2017-06-14 23:03:08 PDT INFO remoteWorkerThread_2: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:03:11 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1 (il_longlive_schema) for region_il'
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_3: SYNC 5000000018 done in 0.005 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_2: SYNC 5000000036 done in 0.005 seconds
2017-06-14 23:03:21 PDT INFO remoteWorkerThread_3: SYNC 5000000019 done in 0.004 seconds
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_2: SYNC 5000000037 done in 0.016 seconds
2017-06-14 23:03:32 PDT INFO remoteWorkerThread_3: SYNC 5000000020 done in 0.012 seconds
2017-06-14 23:03:35 PDT INFO remoteWorkerThread_2: SYNC 5000000038 done in 0.004 seconds
2017-06-14 23:03:48 PDT INFO remoteWorkerThread_2: SYNC 5000000039 done in 0.004 seconds
2017-06-14 23:03:48 PDT INFO remoteWorkerThread_3: SYNC 5000000021 done in 0.005 seconds
2017-06-14 23:03:56 PDT INFO remoteWorkerThread_3: SYNC 5000000022 done in 0.005 seconds
2017-06-14 23:03:57 PDT INFO remoteWorkerThread_2: SYNC 5000000040 done in 0.005 seconds
2017-06-14 23:04:01 PDT INFO remoteWorkerThread_3: SYNC 5000000023 done in 0.005 seconds
2017-06-14 23:04:04 PDT INFO remoteWorkerThread_2: SYNC 5000000041 done in 0.005 seconds
2017-06-14 23:04:17 PDT INFO remoteWorkerThread_2: SYNC 5000000042 done in 0.005 seconds
2017-06-14 23:04:17 PDT INFO remoteWorkerThread_3: SYNC 5000000024 done in 0.005 seconds
2017-06-14 23:04:20 PDT INFO remoteWorkerThread_3: SYNC 5000000025 done in 0.004 seconds
2017-06-14 23:04:23 PDT INFO remoteWorkerThread_2: SYNC 5000000043 done in 0.008 seconds
2017-06-14 23:04:31 PDT INFO remoteWorkerThread_3: SYNC 5000000026 done in 0.005 seconds
2017-06-14 23:04:34 PDT INFO remoteWorkerThread_2: SYNC 5000000044 done in 0.004 seconds
2017-06-14 23:04:42 PDT INFO remoteWorkerThread_3: SYNC 5000000027 done in 0.005 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2498
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2499
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2500
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2526
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: Logswitch to sl_log_2 initiated
CONTEXT: SQL statement "SELECT "_region_il".logswitch_start()"
PL/pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM
2017-06-14 23:04:43 PDT INFO cleanupThread: 0.011 seconds for cleanupEvent()
2017-06-14 23:04:45 PDT INFO remoteWorkerThread_2: SYNC 5000000045 done in 0.004 seconds
2017-06-14 23:04:53 PDT INFO remoteWorkerThread_2: SYNC 5000000046 done in 0.005 seconds
2017-06-14 23:04:53 PDT INFO remoteWorkerThread_3: SYNC 5000000028 done in 0.005 seconds
2017-06-14 23:05:06 PDT INFO remoteWorkerThread_3: SYNC 5000000029 done in 0.005 seconds
2017-06-14 23:05:06 PDT INFO remoteWorkerThread_2: SYNC 5000000047 done in 0.005 seconds
2017-06-14 23:05:19 PDT INFO remoteWorkerThread_2: SYNC 5000000048 done in 0.004 seconds
2017-06-14 23:05:19 PDT INFO remoteWorkerThread_3: SYNC 5000000030 done in 0.012 seconds
2017-06-14 23:05:22 PDT INFO remoteWorkerThread_3: SYNC 5000000031 done in 0.004 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_2: SYNC 5000000049 done in 0.005 seconds
2017-06-14 23:05:38 PDT INFO remoteWorkerThread_2: SYNC 5000000050 done in 0.004 seconds
2017-06-14 23:05:38 PDT INFO remoteWorkerThread_3: SYNC 5000000032 done in 0.005 seconds
2017-06-14 23:05:41 PDT INFO remoteWorkerThread_3: SYNC 5000000033 done in 0.005 seconds
2017-06-14 23:05:45 PDT INFO remoteWorkerThread_2: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:05:53 PDT INFO remoteWorkerThread_3: SYNC 5000000034 done in 0.004 seconds
2017-06-14 23:05:53 PDT INFO remoteWorkerThread_2: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:06:06 PDT INFO remoteWorkerThread_3: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:06:06 PDT INFO remoteWorkerThread_2: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:06:14 PDT INFO remoteWorkerThread_2: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:06:14 PDT INFO remoteWorkerThread_3: SYNC 5000000036 done in 0.006 seconds
2017-06-14 23:06:22 PDT INFO remoteWorkerThread_3: SYNC 5000000037 done in 0.005 seconds
2017-06-14 23:06:25 PDT INFO remoteWorkerThread_2: SYNC 5000000055 done in 0.004 seconds
2017-06-14 23:06:30 PDT INFO remoteWorkerThread_3: SYNC 5000000038 done in 0.005 seconds
2017-06-14 23:06:36 PDT INFO remoteWorkerThread_2: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:06:40 PDT INFO remoteWorkerThread_3: SYNC 5000000039 done in 0.005 seconds
2017-06-14 23:06:48 PDT INFO remoteWorkerThread_2: SYNC 5000000057 done in 0.007 seconds
2017-06-14 23:06:56 PDT INFO remoteWorkerThread_3: SYNC 5000000040 done in 0.004 seconds
2017-06-14 23:06:56 PDT INFO remoteWorkerThread_2: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:07:09 PDT INFO remoteWorkerThread_3: SYNC 5000000041 done in 0.004 seconds
2017-06-14 23:07:09 PDT INFO remoteWorkerThread_2: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:07:10 PDT INFO remoteWorkerThread_3: SYNC 5000000042 done in 0.004 seconds
2017-06-14 23:07:19 PDT INFO remoteWorkerThread_2: SYNC 5000000060 done in 0.005 seconds
2017-06-14 23:07:23 PDT INFO remoteWorkerThread_3: SYNC 5000000043 done in 0.015 seconds
2017-06-14 23:07:23 PDT INFO remoteWorkerThread_2: SYNC 5000000061 done in 0.017 seconds
2017-06-14 23:07:31 PDT INFO remoteWorkerThread_3: SYNC 5000000044 done in 0.004 seconds
2017-06-14 23:07:39 PDT INFO remoteWorkerThread_2: SYNC 5000000062 done in 0.013 seconds
2017-06-14 23:07:42 PDT INFO remoteWorkerThread_3: SYNC 5000000045 done in 0.006 seconds
2017-06-14 23:07:43 PDT INFO remoteWorkerThread_2: SYNC 5000000063 done in 0.005 seconds
2017-06-14 23:07:51 PDT INFO remoteWorkerThread_3: SYNC 5000000046 done in 0.008 seconds
2017-06-14 23:07:56 PDT INFO remoteWorkerThread_2: SYNC 5000000064 done in 0.005 seconds
2017-06-14 23:08:02 PDT INFO remoteWorkerThread_3: SYNC 5000000047 done in 0.022 seconds
2017-06-14 23:08:07 PDT INFO remoteWorkerThread_2: SYNC 5000000065 done in 0.005 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_3: SYNC 5000000048 done in 0.008 seconds
2017-06-14 23:08:18 PDT INFO remoteWorkerThread_2: SYNC 5000000066 done in 0.011 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_3: SYNC 5000000049 done in 0.004 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_2: SYNC 5000000067 done in 0.017 seconds
2017-06-14 23:08:35 PDT INFO remoteWorkerThread_3: SYNC 5000000050 done in 0.005 seconds
2017-06-14 23:08:39 PDT INFO remoteWorkerThread_2: SYNC 5000000068 done in 0.005 seconds
2017-06-14 23:08:40 PDT INFO remoteWorkerThread_3: SYNC 5000000051 done in 0.004 seconds
2017-06-14 23:08:49 PDT INFO remoteWorkerThread_2: SYNC 5000000069 done in 0.005 seconds
2017-06-14 23:08:53 PDT INFO remoteWorkerThread_2: SYNC 5000000070 done in 0.005 seconds
2017-06-14 23:08:54 PDT INFO remoteWorkerThread_3: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:09:02 PDT INFO remoteWorkerThread_3: SYNC 5000000053 done in 0.006 seconds
2017-06-14 23:09:07 PDT INFO remoteWorkerThread_2: SYNC 5000000071 done in 0.005 seconds
2017-06-14 23:09:13 PDT INFO remoteWorkerThread_3: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:09:18 PDT INFO remoteWorkerThread_2: SYNC 5000000072 done in 0.005 seconds
2017-06-14 23:09:24 PDT INFO remoteWorkerThread_3: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:09:29 PDT INFO remoteWorkerThread_2: SYNC 5000000073 done in 0.005 seconds
2017-06-14 23:09:33 PDT INFO remoteWorkerThread_2: SYNC 5000000074 done in 0.005 seconds
2017-06-14 23:09:34 PDT INFO remoteWorkerThread_3: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:09:48 PDT INFO remoteWorkerThread_3: SYNC 5000000057 done in 0.004 seconds
2017-06-14 23:09:48 PDT INFO remoteWorkerThread_2: SYNC 5000000075 done in 0.005 seconds
2017-06-14 23:09:52 PDT INFO remoteWorkerThread_3: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:09:55 PDT INFO remoteWorkerThread_2: SYNC 5000000076 done in 0.004 seconds
2017-06-14 23:10:00 PDT INFO remoteWorkerThread_3: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:10:06 PDT INFO remoteWorkerThread_2: SYNC 5000000077 done in 0.005 seconds
2017-06-14 23:10:11 PDT INFO remoteWorkerThread_3: SYNC 5000000060 done in 0.005 seconds
2017-06-14 23:10:17 PDT INFO remoteWorkerThread_2: SYNC 5000000078 done in 0.005 seconds
2017-06-14 23:10:22 PDT INFO remoteWorkerThread_3: SYNC 5000000061 done in 0.005 seconds
2017-06-14 23:10:28 PDT INFO remoteWorkerThread_2: SYNC 5000000079 done in 0.005 seconds
2017-06-14 23:10:33 PDT INFO remoteWorkerThread_3: SYNC 5000000062 done in 0.005 seconds
2017-06-14 23:10:37 PDT INFO remoteWorkerThread_2: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:10:41 PDT INFO remoteWorkerThread_3: SYNC 5000000063 done in 0.005 seconds
2017-06-14 23:10:46 PDT INFO remoteWorkerThread_2: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:10:52 PDT INFO remoteWorkerThread_3: SYNC 5000000064 done in 0.004 seconds
2017-06-14 23:10:57 PDT INFO remoteWorkerThread_2: SYNC 5000000082 done in 0.004 seconds
2017-06-14 23:11:03 PDT INFO remoteWorkerThread_3: SYNC 5000000065 done in 0.004 seconds
2017-06-14 23:11:08 PDT INFO remoteWorkerThread_2: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:11:14 PDT INFO remoteWorkerThread_3: SYNC 5000000066 done in 0.004 seconds
2017-06-14 23:11:19 PDT INFO remoteWorkerThread_2: SYNC 5000000084 done in 0.005 seconds
2017-06-14 23:11:23 PDT INFO remoteWorkerThread_2: SYNC 5000000085 done in 0.005 seconds
2017-06-14 23:11:25 PDT INFO remoteWorkerThread_3: SYNC 5000000067 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_2: SYNC 5000000086 done in 0.006 seconds
2017-06-14 23:11:38 PDT INFO remoteWorkerThread_3: SYNC 5000000068 done in 0.005 seconds
2017-06-14 23:11:42 PDT INFO remoteWorkerThread_3: SYNC 5000000069 done in 0.005 seconds
2017-06-14 23:11:46 PDT INFO remoteWorkerThread_2: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: connection for provider 3 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:54 PDT INFO remoteWorkerThread_3: SYNC 5000000070 done in 0.005 seconds
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: connection for provider 2 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 2
2017-06-14 23:11:58 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:58 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:11:58 PDT INFO remoteWorkerThread_2: SYNC 5000000088 done in 0.012 seconds
2017-06-14 23:12:02 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:12:02 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:02 PDT INFO remoteWorkerThread_3: SYNC 5000000071 done in 0.020 seconds
2017-06-14 23:12:06 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:06 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:07 PDT INFO remoteWorkerThread_2: SYNC 5000000089 done in 0.278 seconds
2017-06-14 23:12:13 PDT INFO remoteWorkerThread_3: SYNC 5000000072 done in 0.003 seconds
2017-06-14 23:12:24 PDT INFO remoteWorkerThread_3: SYNC 5000000073 done in 0.003 seconds
2017-06-14 23:12:33 PDT INFO remoteWorkerThread_2: SYNC 5000000090 done in 0.005 seconds
2017-06-14 23:12:37 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:12:37 PDT INFO remoteWorkerThread_2: SYNC 5000000091 done in 0.004 seconds
2017-06-14 23:12:37 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:38 PDT INFO remoteWorkerThread_3: SYNC 5000000074 done in 0.522 seconds
2017-06-14 23:12:41 PDT INFO remoteWorkerThread_3: SYNC 5000000075 done in 0.005 seconds
2017-06-14 23:12:52 PDT INFO remoteWorkerThread_3: SYNC 5000000076 done in 0.005 seconds
2017-06-14 23:12:55 PDT INFO remoteWorkerThread_2: SYNC 5000000092 done in 0.005 seconds
2017-06-14 23:13:03 PDT INFO remoteWorkerThread_2: SYNC 5000000093 done in 0.016 seconds
2017-06-14 23:13:09 PDT INFO remoteWorkerThread_3: SYNC 5000000077 done in 0.005 seconds
2017-06-14 23:13:10 PDT INFO remoteWorkerThread_3: SYNC 5000000078 done in 0.005 seconds
2017-06-14 23:13:14 PDT INFO remoteWorkerThread_2: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:13:21 PDT INFO remoteWorkerThread_3: SYNC 5000000079 done in 0.005 seconds
2017-06-14 23:13:25 PDT INFO remoteWorkerThread_2: SYNC 5000000095 done in 0.004 seconds
2017-06-14 23:13:32 PDT INFO remoteWorkerThread_3: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:13:36 PDT INFO remoteWorkerThread_2: SYNC 5000000096 done in 0.012 seconds
2017-06-14 23:13:43 PDT INFO remoteWorkerThread_3: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:13:47 PDT INFO remoteWorkerThread_2: SYNC 5000000097 done in 0.089 seconds
2017-06-14 23:13:54 PDT INFO remoteWorkerThread_3: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:13:58 PDT INFO remoteWorkerThread_2: SYNC 5000000098 done in 0.006 seconds
2017-06-14 23:14:06 PDT INFO remoteWorkerThread_3: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:14:10 PDT INFO remoteWorkerThread_3: SYNC 5000000084 done in 0.004 seconds
2017-06-14 23:14:11 PDT INFO remoteWorkerThread_2: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:14:21 PDT INFO remoteWorkerThread_3: SYNC 5000000085 done in 0.015 seconds
2017-06-14 23:14:21 PDT INFO remoteWorkerThread_2: SYNC 5000000100 done in 0.016 seconds
2017-06-14 23:14:32 PDT INFO remoteWorkerThread_3: SYNC 5000000086 done in 0.009 seconds
2017-06-14 23:14:35 PDT INFO remoteWorkerThread_2: SYNC 5000000101 done in 0.005 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2529
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2552
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2567
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2568
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1
CONTEXT: PL/pgSQL function _region_il.cleanupevent(interval) line 95 at assignment
2017-06-14 23:14:43 PDT INFO cleanupThread: 0.086 seconds for cleanupEvent()
2017-06-14 23:14:49 PDT INFO remoteWorkerThread_3: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_3: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 3 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 3
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: connection for provider 2 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:54 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:54 PDT INFO remoteWorkerThread_2: SYNC 5000000102 done in 0.037 seconds
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:15:02 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:15:02 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:02 PDT INFO remoteWorkerThread_2: SYNC 5000000103 done in 0.013 seconds
2017-06-14 23:15:06 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:06 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:06 PDT INFO remoteWorkerThread_3: SYNC 5000000089 done in 0.013 seconds
2017-06-14 23:15:14 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:14 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:17 PDT INFO remoteWorkerThread_3: SYNC 5000000090 done in 2.438 seconds
2017-06-14 23:15:20 PDT INFO remoteWorkerThread_2: SYNC 5000000104 done in 0.003 seconds
2017-06-14 23:15:21 PDT INFO remoteWorkerThread_2: SYNC 5000000105 done in 0.002 seconds
2017-06-14 23:15:25 PDT INFO remoteWorkerThread_3: SYNC 5000000091 done in 0.016 seconds
2017-06-14 23:15:32 PDT INFO remoteWorkerThread_2: SYNC 5000000106 done in 0.002 seconds
2017-06-14 23:15:36 PDT INFO remoteWorkerThread_3: SYNC 5000000092 done in 0.005 seconds
2017-06-14 23:15:43 PDT INFO remoteWorkerThread_2: SYNC 5000000107 done in 0.003 seconds
2017-06-14 23:15:47 PDT INFO remoteWorkerThread_3: SYNC 5000000093 done in 0.006 seconds
2017-06-14 23:15:55 PDT INFO remoteWorkerThread_2: SYNC 5000000108 done in 0.003 seconds
2017-06-14 23:15:55 PDT INFO remoteWorkerThread_3: SYNC 5000000094 done in 0.013 seconds
2017-06-14 23:16:06 PDT INFO remoteWorkerThread_2: SYNC 5000000109 done in 0.003 seconds
2017-06-14 23:16:10 PDT INFO remoteWorkerThread_3: SYNC 5000000095 done in 0.004 seconds
2017-06-14 23:16:11 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:11 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:11 PDT INFO remoteWorkerThread_2: SYNC 5000000110 done in 0.058 seconds
2017-06-14 23:16:12 PDT INFO remoteWorkerThread_3: SYNC 5000000096 done in 0.012 seconds
2017-06-14 23:16:27 PDT INFO remoteWorkerThread_2: SYNC 5000000111 done in 0.008 seconds
2017-06-14 23:16:31 PDT INFO remoteWorkerThread_2: SYNC 5000000112 done in 0.013 seconds
2017-06-14 23:16:31 PDT INFO remoteWorkerThread_3: SYNC 5000000097 done in 0.037 seconds
2017-06-14 23:16:42 PDT INFO remoteWorkerThread_3: SYNC 5000000098 done in 0.006 seconds
2017-06-14 23:16:42 PDT INFO remoteWorkerThread_2: SYNC 5000000113 done in 0.006 seconds
2017-06-14 23:16:53 PDT INFO remoteWorkerThread_3: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:16:53 PDT INFO remoteWorkerThread_2: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:17:04 PDT INFO remoteWorkerThread_3: SYNC 5000000100 done in 0.005 seconds
2017-06-14 23:17:04 PDT INFO remoteWorkerThread_2: SYNC 5000000115 done in 0.006 seconds
2017-06-14 23:17:08 PDT INFO remoteWorkerThread_3: SYNC 5000000101 done in 0.005 seconds
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_2: SYNC 5000000116 done in 0.004 seconds
2017-06-14 23:17:19 PDT INFO remoteWorkerThread_3: SYNC 5000000102 done in 0.005 seconds
2017-06-14 23:17:25 PDT INFO remoteWorkerThread_2: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:17:30 PDT INFO remoteWorkerThread_3: SYNC 5000000103 done in 0.005 seconds
2017-06-14 23:17:34 PDT INFO remoteWorkerThread_2: SYNC 5000000118 done in 0.013 seconds
2017-06-14 23:17:38 PDT INFO remoteWorkerThread_3: SYNC 5000000104 done in 0.012 seconds
2017-06-14 23:17:43 PDT INFO remoteWorkerThread_2: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:17:49 PDT INFO remoteWorkerThread_3: SYNC 5000000105 done in 0.004 seconds
2017-06-14 23:17:53 PDT INFO remoteWorkerThread_2: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_2: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_3: SYNC 5000000106 done in 0.006 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_2: SYNC 5000000122 done in 0.005 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_3: SYNC 5000000107 done in 0.006 seconds
2017-06-14 23:18:27 PDT INFO remoteWorkerThread_2: SYNC 5000000123 done in 0.005 seconds
2017-06-14 23:18:27 PDT INFO remoteWorkerThread_3: SYNC 5000000108 done in 0.006 seconds
2017-06-14 23:18:30 PDT INFO remoteWorkerThread_3: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:18:33 PDT INFO remoteWorkerThread_2: SYNC 5000000124 done in 0.004 seconds
2017-06-14 23:18:38 PDT INFO remoteWorkerThread_3: SYNC 5000000110 done in 0.004 seconds
2017-06-14 23:18:44 PDT INFO remoteWorkerThread_2: SYNC 5000000125 done in 0.004 seconds
2017-06-14 23:18:49 PDT INFO remoteWorkerThread_3: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:18:55 PDT INFO remoteWorkerThread_2: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:19:00 PDT INFO remoteWorkerThread_3: SYNC 5000000112 done in 0.011 seconds
2017-06-14 23:19:04 PDT INFO remoteWorkerThread_2: SYNC 5000000127 done in 0.005 seconds
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_3: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:19:12 PDT INFO remoteWorkerThread_2: SYNC 5000000128 done in 0.005 seconds
2017-06-14 23:19:20 PDT INFO remoteWorkerThread_3: SYNC 5000000114 done in 0.004 seconds
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_2: SYNC 5000000129 done in 0.004 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_3: SYNC 5000000115 done in 0.005 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_2: SYNC 5000000130 done in 0.006 seconds
2017-06-14 23:19:47 PDT INFO remoteWorkerThread_2: SYNC 5000000131 done in 0.005 seconds
2017-06-14 23:19:47 PDT INFO remoteWorkerThread_3: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_3: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:19:51 PDT INFO remoteWorkerThread_2: SYNC 5000000132 done in 0.005 seconds
2017-06-14 23:20:02 PDT INFO remoteWorkerThread_3: SYNC 5000000118 done in 0.005 seconds
2017-06-14 23:20:02 PDT INFO remoteWorkerThread_2: SYNC 5000000133 done in 0.006 seconds
2017-06-14 23:20:16 PDT INFO remoteWorkerThread_3: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:20:16 PDT INFO remoteWorkerThread_2: SYNC 5000000134 done in 0.006 seconds
2017-06-14 23:20:19 PDT INFO remoteWorkerThread_3: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:20:22 PDT INFO remoteWorkerThread_2: SYNC 5000000135 done in 0.005 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_3: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_2: SYNC 5000000136 done in 0.006 seconds
2017-06-14 23:20:38 PDT INFO remoteWorkerThread_3: SYNC 5000000122 done in 0.005 seconds
2017-06-14 23:20:41 PDT INFO remoteWorkerThread_2: SYNC 5000000137 done in 0.005 seconds

辅节点1查看Slony-I日志信息

$ cat /usr/local/pgsql/data/pg_log/node2/region_il-2017-06-14.log
2017-06-14 22:57:41 PDT CONFIG main: slon version 2.2.4 starting up
2017-06-14 22:57:41 PDT INFO slon: watchdog process started
2017-06-14 22:57:41 PDT CONFIG slon: watchdog ready - pid = 15535
2017-06-14 22:57:41 PDT CONFIG slon: worker process created - pid = 15536
2017-06-14 22:57:41 PDT CONFIG main: Integer option vac_frequency = 3
2017-06-14 22:57:41 PDT CONFIG main: Integer option log_level = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_interval = 1000
2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_interval_timeout = 10000
2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_group_maxsize = 20
2017-06-14 22:57:41 PDT CONFIG main: Integer option syslog = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option quit_sync_provider = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option remote_listen_timeout = 300
2017-06-14 22:57:41 PDT CONFIG main: Integer option monitor_interval = 500
2017-06-14 22:57:41 PDT CONFIG main: Integer option explain_interval = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_idle = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_interval = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_count = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option apply_cache_size = 100
2017-06-14 22:57:41 PDT CONFIG main: Boolean option log_pid = 0
2017-06-14 22:57:41 PDT CONFIG main: Boolean option log_timestamp = 1
2017-06-14 22:57:41 PDT CONFIG main: Boolean option tcp_keepalive = 1
2017-06-14 22:57:41 PDT CONFIG main: Boolean option monitor_threads = 1
2017-06-14 22:57:41 PDT CONFIG main: Real option real_placeholder = 0.000000
2017-06-14 22:57:41 PDT CONFIG main: String option cluster_name = region_il
2017-06-14 22:57:41 PDT CONFIG main: String option conn_info = host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888
2017-06-14 22:57:41 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node2.pid
2017-06-14 22:57:41 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2017-06-14 22:57:41 PDT CONFIG main: String option archive_dir = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option sql_on_connection = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option lag_interval = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option command_on_logarchive = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option syslog_facility = LOCAL0
2017-06-14 22:57:41 PDT CONFIG main: String option syslog_ident = slon
2017-06-14 22:57:41 PDT CONFIG main: String option cleanup_interval = 10 minutes
2017-06-14 22:57:41 PDT CONFIG main: local node id = 2
2017-06-14 22:57:41 PDT INFO main: main process started
2017-06-14 22:57:41 PDT CONFIG main: launching sched_start_mainloop
2017-06-14 22:57:41 PDT CONFIG main: loading current cluster configuration
2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=1 no_comment='Node 1 - region_il@10.189.102.118'
2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=3 no_comment='<event pending>'
2017-06-14 22:57:41 PDT CONFIG storePath: pa_server=1 pa_client=2 pa_conninfo="host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:57:41 PDT CONFIG storePath: pa_server=3 pa_client=2 pa_conninfo="host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG main: last local event sequence = 5000000002
2017-06-14 22:57:41 PDT CONFIG main: configuration complete - starting threads
2017-06-14 22:57:41 PDT INFO localListenThread: thread starts
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG enableNode: no_id=1
2017-06-14 22:57:41 PDT INFO main: running scheduler mainloop
2017-06-14 22:57:41 PDT CONFIG cleanupThread: thread starts
2017-06-14 22:57:41 PDT INFO syncThread: thread starts
2017-06-14 22:57:41 PDT INFO monitorThread: thread starts
2017-06-14 22:57:41 PDT INFO remoteListenThread_1: thread starts
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: thread starts
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG cleanupThread: bias = 60
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=3 no_comment='Node 3 - region_il@10.189.100.226' - update node
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG enableNode: no_id=3
2017-06-14 22:57:41 PDT INFO remoteListenThread_3: thread starts
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_3: thread starts
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000008 done in 0.015 seconds
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000010 done in 0.002 seconds
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000014 done in 0.011 seconds
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000022 done in 0.012 seconds
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000025 done in 0.002 seconds
2017-06-14 22:57:45 PDT INFO remoteWorkerThread_1: SYNC 5000000026 done in 0.003 seconds
2017-06-14 22:57:56 PDT INFO remoteWorkerThread_1: SYNC 5000000027 done in 0.003 seconds
2017-06-14 22:58:10 PDT INFO remoteWorkerThread_1: SYNC 5000000028 done in 0.003 seconds
2017-06-14 22:58:14 PDT INFO remoteWorkerThread_1: SYNC 5000000029 done in 0.003 seconds
2017-06-14 22:58:28 PDT INFO remoteWorkerThread_1: SYNC 5000000030 done in 0.003 seconds
2017-06-14 22:58:39 PDT INFO remoteWorkerThread_1: SYNC 5000000031 done in 0.003 seconds
2017-06-14 22:58:43 PDT INFO remoteWorkerThread_1: SYNC 5000000032 done in 0.003 seconds
2017-06-14 22:59:01 PDT INFO remoteWorkerThread_1: SYNC 5000000033 done in 0.003 seconds
2017-06-14 22:59:05 PDT INFO remoteWorkerThread_1: SYNC 5000000034 done in 0.003 seconds
2017-06-14 22:59:19 PDT INFO remoteWorkerThread_1: SYNC 5000000035 done in 0.003 seconds
2017-06-14 22:59:23 PDT INFO remoteWorkerThread_1: SYNC 5000000036 done in 0.002 seconds
2017-06-14 22:59:41 PDT INFO remoteWorkerThread_1: SYNC 5000000037 done in 0.003 seconds
2017-06-14 22:59:45 PDT INFO remoteWorkerThread_1: SYNC 5000000038 done in 0.002 seconds
2017-06-14 22:59:59 PDT INFO remoteWorkerThread_1: SYNC 5000000039 done in 0.003 seconds
2017-06-14 23:00:03 PDT INFO remoteWorkerThread_1: SYNC 5000000040 done in 0.003 seconds
2017-06-14 23:00:21 PDT INFO remoteWorkerThread_1: SYNC 5000000041 done in 0.003 seconds
2017-06-14 23:00:25 PDT INFO remoteWorkerThread_1: SYNC 5000000042 done in 0.002 seconds
2017-06-14 23:00:39 PDT INFO remoteWorkerThread_1: SYNC 5000000043 done in 0.003 seconds
2017-06-14 23:00:43 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:00:43 PDT INFO remoteWorkerThread_1: SYNC 5000000044 done in 0.003 seconds
2017-06-14 23:00:43 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:43 PDT INFO remoteWorkerThread_3: SYNC 5000000003 done in 0.014 seconds
2017-06-14 23:00:56 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:00:56 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:56 PDT INFO remoteWorkerThread_3: SYNC 5000000004 done in 0.022 seconds
2017-06-14 23:01:00 PDT INFO remoteWorkerThread_3: SYNC 5000000005 done in 0.013 seconds
2017-06-14 23:01:01 PDT INFO remoteWorkerThread_1: SYNC 5000000045 done in 0.003 seconds
2017-06-14 23:01:04 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:01:04 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:01:04 PDT INFO remoteWorkerThread_1: SYNC 5000000046 done in 0.014 seconds
2017-06-14 23:01:15 PDT INFO remoteWorkerThread_3: SYNC 5000000006 done in 0.005 seconds
2017-06-14 23:01:15 PDT INFO remoteWorkerThread_1: SYNC 5000000047 done in 0.006 seconds
2017-06-14 23:01:19 PDT INFO remoteWorkerThread_3: SYNC 5000000007 done in 0.004 seconds
2017-06-14 23:01:26 PDT INFO remoteWorkerThread_1: SYNC 5000000048 done in 0.005 seconds
2017-06-14 23:01:30 PDT INFO remoteWorkerThread_3: SYNC 5000000008 done in 0.005 seconds
2017-06-14 23:01:37 PDT INFO remoteWorkerThread_1: SYNC 5000000049 done in 0.005 seconds
2017-06-14 23:01:41 PDT INFO remoteWorkerThread_3: SYNC 5000000009 done in 0.005 seconds
2017-06-14 23:01:49 PDT INFO remoteWorkerThread_1: SYNC 5000000050 done in 0.013 seconds
2017-06-14 23:01:53 PDT INFO remoteWorkerThread_3: SYNC 5000000010 done in 0.005 seconds
2017-06-14 23:01:53 PDT INFO remoteWorkerThread_1: SYNC 5000000051 done in 0.006 seconds
2017-06-14 23:02:01 PDT INFO remoteWorkerThread_3: SYNC 5000000011 done in 0.006 seconds
2017-06-14 23:02:07 PDT INFO remoteWorkerThread_1: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:02:11 PDT INFO remoteWorkerThread_3: SYNC 5000000012 done in 0.005 seconds
2017-06-14 23:02:15 PDT INFO remoteWorkerThread_1: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:02:19 PDT INFO remoteWorkerThread_3: SYNC 5000000013 done in 0.005 seconds
2017-06-14 23:02:26 PDT INFO remoteWorkerThread_1: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:02:30 PDT INFO remoteWorkerThread_3: SYNC 5000000014 done in 0.005 seconds
2017-06-14 23:02:37 PDT INFO remoteWorkerThread_1: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:02:41 PDT INFO remoteWorkerThread_3: SYNC 5000000015 done in 0.005 seconds
2017-06-14 23:02:45 PDT INFO remoteWorkerThread_1: SYNC 5000000056 done in 0.006 seconds
2017-06-14 23:02:49 PDT INFO remoteWorkerThread_3: SYNC 5000000016 done in 0.006 seconds
2017-06-14 23:02:55 PDT INFO remoteWorkerThread_1: SYNC 5000000057 done in 0.005 seconds
2017-06-14 23:03:00 PDT INFO remoteWorkerThread_3: SYNC 5000000017 done in 0.005 seconds
2017-06-14 23:03:06 PDT INFO remoteWorkerThread_1: SYNC 5000000058 done in 0.009 seconds
2017-06-14 23:03:11 PDT INFO remoteWorkerThread_3: SYNC 5000000018 done in 0.004 seconds
2017-06-14 23:03:17 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1 (il_longlive_schema) for region_il'
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:17 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000060 done in 0.014 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000062 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000064 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000066 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000068 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000070 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000072 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000074 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000076 done in 0.002 seconds
2017-06-14 23:03:21 PDT INFO remoteWorkerThread_3: SYNC 5000000019 done in 0.005 seconds
2017-06-14 23:03:25 PDT INFO remoteWorkerThread_1: SYNC 5000000077 done in 0.002 seconds
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_3: SYNC 5000000020 done in 0.005 seconds
2017-06-14 23:03:36 PDT INFO remoteWorkerThread_1: SYNC 5000000078 done in 0.003 seconds
2017-06-14 23:03:40 PDT INFO remoteWorkerThread_3: SYNC 5000000021 done in 0.005 seconds
2017-06-14 23:03:44 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:44 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:44 PDT INFO remoteWorkerThread_1: SYNC 5000000079 done in 0.015 seconds
2017-06-14 23:03:54 PDT INFO remoteWorkerThread_1: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:03:58 PDT INFO remoteWorkerThread_3: SYNC 5000000022 done in 0.005 seconds
2017-06-14 23:04:01 PDT INFO remoteWorkerThread_3: SYNC 5000000023 done in 0.005 seconds
2017-06-14 23:04:06 PDT INFO remoteWorkerThread_1: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:04:12 PDT INFO remoteWorkerThread_3: SYNC 5000000024 done in 0.005 seconds
2017-06-14 23:04:16 PDT INFO remoteWorkerThread_1: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:04:20 PDT INFO remoteWorkerThread_3: SYNC 5000000025 done in 0.005 seconds
2017-06-14 23:04:24 PDT INFO remoteWorkerThread_1: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:04:32 PDT INFO remoteWorkerThread_3: SYNC 5000000026 done in 0.005 seconds
2017-06-14 23:04:33 PDT INFO remoteWorkerThread_1: SYNC 5000000084 done in 0.004 seconds
2017-06-14 23:04:44 PDT INFO remoteWorkerThread_3: SYNC 5000000027 done in 0.005 seconds
2017-06-14 23:04:44 PDT INFO remoteWorkerThread_1: SYNC 5000000085 done in 0.006 seconds
2017-06-14 23:04:54 PDT INFO remoteWorkerThread_3: SYNC 5000000028 done in 0.005 seconds
2017-06-14 23:04:58 PDT INFO remoteWorkerThread_1: SYNC 5000000086 done in 0.005 seconds
2017-06-14 23:05:01 PDT INFO remoteWorkerThread_3: SYNC 5000000029 done in 0.006 seconds
2017-06-14 23:05:05 PDT INFO remoteWorkerThread_1: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:05:13 PDT INFO remoteWorkerThread_3: SYNC 5000000030 done in 0.005 seconds
2017-06-14 23:05:14 PDT INFO remoteWorkerThread_1: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_3: SYNC 5000000031 done in 0.005 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_1: SYNC 5000000089 done in 0.006 seconds
2017-06-14 23:05:29 PDT INFO remoteWorkerThread_3: SYNC 5000000032 done in 0.005 seconds
2017-06-14 23:05:36 PDT INFO remoteWorkerThread_1: SYNC 5000000090 done in 0.006 seconds
2017-06-14 23:05:40 PDT INFO remoteWorkerThread_3: SYNC 5000000033 done in 0.004 seconds
2017-06-14 23:05:47 PDT INFO remoteWorkerThread_1: SYNC 5000000091 done in 0.005 seconds
2017-06-14 23:05:51 PDT INFO remoteWorkerThread_3: SYNC 5000000034 done in 0.004 seconds
2017-06-14 23:05:58 PDT INFO remoteWorkerThread_1: SYNC 5000000092 done in 0.005 seconds
2017-06-14 23:06:02 PDT INFO remoteWorkerThread_3: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:06:09 PDT INFO remoteWorkerThread_1: SYNC 5000000093 done in 0.005 seconds
2017-06-14 23:06:13 PDT INFO remoteWorkerThread_1: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:06:13 PDT INFO remoteWorkerThread_3: SYNC 5000000036 done in 0.004 seconds
2017-06-14 23:06:21 PDT INFO remoteWorkerThread_3: SYNC 5000000037 done in 0.004 seconds
2017-06-14 23:06:27 PDT INFO remoteWorkerThread_1: SYNC 5000000095 done in 0.004 seconds
2017-06-14 23:06:31 PDT INFO remoteWorkerThread_3: SYNC 5000000038 done in 0.005 seconds
2017-06-14 23:06:35 PDT INFO remoteWorkerThread_1: SYNC 5000000096 done in 0.005 seconds
2017-06-14 23:06:39 PDT INFO remoteWorkerThread_3: SYNC 5000000039 done in 0.004 seconds
2017-06-14 23:06:46 PDT INFO remoteWorkerThread_1: SYNC 5000000097 done in 0.005 seconds
2017-06-14 23:06:50 PDT INFO remoteWorkerThread_3: SYNC 5000000040 done in 0.004 seconds
2017-06-14 23:06:57 PDT INFO remoteWorkerThread_1: SYNC 5000000098 done in 0.005 seconds
2017-06-14 23:07:01 PDT INFO remoteWorkerThread_3: SYNC 5000000041 done in 0.005 seconds
2017-06-14 23:07:08 PDT INFO remoteWorkerThread_1: SYNC 5000000099 done in 0.004 seconds
2017-06-14 23:07:12 PDT INFO remoteWorkerThread_3: SYNC 5000000042 done in 0.005 seconds
2017-06-14 23:07:13 PDT INFO remoteWorkerThread_1: SYNC 5000000100 done in 0.004 seconds
2017-06-14 23:07:22 PDT INFO remoteWorkerThread_3: SYNC 5000000043 done in 0.005 seconds
2017-06-14 23:07:27 PDT INFO remoteWorkerThread_1: SYNC 5000000101 done in 0.030 seconds
2017-06-14 23:07:31 PDT INFO remoteWorkerThread_3: SYNC 5000000044 done in 0.005 seconds
2017-06-14 23:07:35 PDT INFO remoteWorkerThread_1: SYNC 5000000102 done in 0.004 seconds
2017-06-14 23:07:40 PDT INFO remoteWorkerThread_3: SYNC 5000000045 done in 0.009 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15654
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15655
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15658
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: Logswitch to sl_log_2 initiated
CONTEXT: SQL statement "SELECT "_region_il".logswitch_start()"
PL/pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM
2017-06-14 23:07:42 PDT INFO cleanupThread: 0.019 seconds for cleanupEvent()
2017-06-14 23:07:46 PDT INFO remoteWorkerThread_1: SYNC 5000000103 done in 0.009 seconds
2017-06-14 23:07:51 PDT INFO remoteWorkerThread_3: SYNC 5000000046 done in 0.005 seconds
2017-06-14 23:07:57 PDT INFO remoteWorkerThread_1: SYNC 5000000104 done in 0.010 seconds
2017-06-14 23:08:02 PDT INFO remoteWorkerThread_3: SYNC 5000000047 done in 0.009 seconds
2017-06-14 23:08:08 PDT INFO remoteWorkerThread_1: SYNC 5000000105 done in 0.012 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_1: SYNC 5000000106 done in 0.008 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_3: SYNC 5000000048 done in 0.013 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_3: SYNC 5000000049 done in 0.009 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_1: SYNC 5000000107 done in 0.010 seconds
2017-06-14 23:08:35 PDT INFO remoteWorkerThread_3: SYNC 5000000050 done in 0.005 seconds
2017-06-14 23:08:39 PDT INFO remoteWorkerThread_1: SYNC 5000000108 done in 0.006 seconds
2017-06-14 23:08:39 PDT INFO remoteWorkerThread_3: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:08:43 PDT INFO remoteWorkerThread_1: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:08:54 PDT INFO remoteWorkerThread_3: SYNC 5000000052 done in 0.006 seconds
2017-06-14 23:08:54 PDT INFO remoteWorkerThread_1: SYNC 5000000110 done in 0.006 seconds
2017-06-14 23:09:04 PDT INFO remoteWorkerThread_3: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:09:08 PDT INFO remoteWorkerThread_1: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:09:11 PDT INFO remoteWorkerThread_3: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:09:16 PDT INFO remoteWorkerThread_1: SYNC 5000000112 done in 0.005 seconds
2017-06-14 23:09:22 PDT INFO remoteWorkerThread_3: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:09:27 PDT INFO remoteWorkerThread_1: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:09:33 PDT INFO remoteWorkerThread_3: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:09:34 PDT INFO remoteWorkerThread_1: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:09:45 PDT INFO remoteWorkerThread_1: SYNC 5000000115 done in 0.005 seconds
2017-06-14 23:09:48 PDT INFO remoteWorkerThread_3: SYNC 5000000057 done in 0.005 seconds
2017-06-14 23:09:49 PDT INFO remoteWorkerThread_3: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:09:59 PDT INFO remoteWorkerThread_1: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:10:00 PDT INFO remoteWorkerThread_3: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:10:04 PDT INFO remoteWorkerThread_1: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:10:10 PDT INFO remoteWorkerThread_3: SYNC 5000000060 done in 0.005 seconds
2017-06-14 23:10:14 PDT INFO remoteWorkerThread_1: SYNC 5000000118 done in 0.005 seconds
2017-06-14 23:10:22 PDT INFO remoteWorkerThread_3: SYNC 5000000061 done in 0.005 seconds
2017-06-14 23:10:28 PDT INFO remoteWorkerThread_1: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:10:33 PDT INFO remoteWorkerThread_3: SYNC 5000000062 done in 0.005 seconds
2017-06-14 23:10:39 PDT INFO remoteWorkerThread_1: SYNC 5000000120 done in 0.004 seconds
2017-06-14 23:10:43 PDT INFO remoteWorkerThread_3: SYNC 5000000063 done in 0.005 seconds
2017-06-14 23:10:43 PDT INFO remoteWorkerThread_1: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:10:52 PDT INFO remoteWorkerThread_3: SYNC 5000000064 done in 0.005 seconds
2017-06-14 23:10:57 PDT INFO remoteWorkerThread_1: SYNC 5000000122 done in 0.004 seconds
2017-06-14 23:11:03 PDT INFO remoteWorkerThread_3: SYNC 5000000065 done in 0.005 seconds
2017-06-14 23:11:08 PDT INFO remoteWorkerThread_1: SYNC 5000000123 done in 0.005 seconds
2017-06-14 23:11:14 PDT INFO remoteWorkerThread_3: SYNC 5000000066 done in 0.005 seconds
2017-06-14 23:11:19 PDT INFO remoteWorkerThread_1: SYNC 5000000124 done in 0.005 seconds
2017-06-14 23:11:25 PDT INFO remoteWorkerThread_3: SYNC 5000000067 done in 0.005 seconds
2017-06-14 23:11:30 PDT INFO remoteWorkerThread_1: SYNC 5000000125 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_1: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:11:36 PDT INFO remoteWorkerThread_3: SYNC 5000000068 done in 0.005 seconds
2017-06-14 23:11:40 PDT INFO remoteWorkerThread_3: SYNC 5000000069 done in 0.004 seconds
2017-06-14 23:11:45 PDT INFO remoteWorkerThread_1: SYNC 5000000127 done in 0.005 seconds
2017-06-14 23:11:51 PDT INFO remoteWorkerThread_3: SYNC 5000000070 done in 0.004 seconds
2017-06-14 23:11:56 PDT CONFIG storeSubscribe: sub_set=1 sub_provider=1 sub_forward='t'
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:11:56 PDT INFO copy_set 1 - omit=f - bool=0
2017-06-14 23:11:56 PDT INFO omit is FALSE
2017-06-14 23:11:56 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connected to provider DB
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity_result"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result_logbook"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: all tables for set 1 found on subscriber
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_entity_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_result_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_logbook_record_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity"
2017-06-14 23:11:57 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity"
NOTICE: truncate of "il_longlive"."entity" succeeded
2017-06-14 23:12:02 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:12:02 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:02 PDT INFO remoteWorkerThread_3: SYNC 5000000071 done in 0.029 seconds
2017-06-14 23:12:13 PDT INFO remoteWorkerThread_3: SYNC 5000000072 done in 0.003 seconds
2017-06-14 23:12:28 PDT INFO remoteWorkerThread_3: SYNC 5000000073 done in 0.016 seconds
2017-06-14 23:12:33 PDT CONFIG remoteWorkerThread_1: 125045870 bytes copied for table "il_longlive"."entity"
2017-06-14 23:12:35 PDT INFO remoteWorkerThread_3: SYNC 5000000074 done in 0.010 seconds
2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: 46.468 seconds to copy table "il_longlive"."entity"
2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity_result"
2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity_result"
NOTICE: truncate of "il_longlive"."entity_result" succeeded
2017-06-14 23:12:46 PDT INFO remoteWorkerThread_3: SYNC 5000000075 done in 0.003 seconds
2017-06-14 23:12:50 PDT INFO remoteWorkerThread_3: SYNC 5000000076 done in 0.003 seconds
2017-06-14 23:13:12 PDT CONFIG remoteWorkerThread_1: 66996342 bytes copied for table "il_longlive"."entity_result"
2017-06-14 23:13:14 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:13:14 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:13:14 PDT INFO remoteWorkerThread_3: SYNC 5000000078 done in 0.025 seconds
2017-06-14 23:13:23 PDT INFO remoteWorkerThread_3: SYNC 5000000079 done in 0.013 seconds
2017-06-14 23:13:32 PDT INFO remoteWorkerThread_3: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:13:41 PDT INFO remoteWorkerThread_3: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:13:55 PDT INFO remoteWorkerThread_3: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: 74.255 seconds to copy table "il_longlive"."entity_result"
2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result"
2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result"
NOTICE: truncate of "il_longlive"."result" succeeded
2017-06-14 23:14:07 PDT INFO remoteWorkerThread_3: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:14:20 PDT INFO remoteWorkerThread_3: SYNC 5000000084 done in 0.020 seconds
2017-06-14 23:14:23 PDT INFO remoteWorkerThread_3: SYNC 5000000085 done in 0.005 seconds
2017-06-14 23:14:29 PDT CONFIG remoteWorkerThread_1: 105961503 bytes copied for table "il_longlive"."result"
2017-06-14 23:14:33 PDT INFO remoteWorkerThread_3: SYNC 5000000086 done in 0.005 seconds
2017-06-14 23:14:41 PDT INFO remoteWorkerThread_3: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 52.781 seconds to copy table "il_longlive"."result"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result_logbook"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result_logbook"
NOTICE: truncate of "il_longlive"."result_logbook" succeeded
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 79 bytes copied for table "il_longlive"."result_logbook"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 0.322 seconds to copy table "il_longlive"."result_logbook"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 103 ("il_longlive"."result_id_seq") to 2611527
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 102 ("il_longlive"."entity_result_id_seq") to 2611527
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 104 ("il_longlive"."result_logbook_record_id_seq") to 1
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 101 ("il_longlive"."entity_entity_id_seq") to 2528612
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: 0.002 seconds to copy sequences
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: copy_set SYNC found, use event seqno 5000000130.
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: 0.005 seconds to build initial setsync status
2017-06-14 23:14:50 PDT INFO copy_set 1 done in 173.913 seconds
2017-06-14 23:14:50 PDT CONFIG enableSubscription: sub_set=1
2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1
2017-06-14 23:14:50 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000131 done in 0.053 seconds
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:50 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000135 done in 0.016 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000143 done in 0.005 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000147 done in 0.005 seconds
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3
2017-06-14 23:14:52 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:52 PDT INFO remoteWorkerThread_3: SYNC 5000000088 done in 0.013 seconds
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1
2017-06-14 23:15:00 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:00 PDT INFO remoteWorkerThread_1: SYNC 5000000150 done in 0.006 seconds
2017-06-14 23:15:05 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:05 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:05 PDT INFO remoteWorkerThread_3: SYNC 5000000089 done in 0.054 seconds
2017-06-14 23:15:11 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:11 PDT INFO remoteWorkerThread_1: SYNC 5000000151 done in 0.004 seconds
2017-06-14 23:15:15 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:15 PDT INFO remoteWorkerThread_1: SYNC 5000000152 done in 0.004 seconds
2017-06-14 23:15:16 PDT INFO remoteWorkerThread_3: SYNC 5000000090 done in 0.005 seconds
2017-06-14 23:15:26 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:26 PDT INFO remoteWorkerThread_1: SYNC 5000000153 done in 0.007 seconds
2017-06-14 23:15:26 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:26 PDT INFO remoteWorkerThread_3: SYNC 5000000091 done in 0.025 seconds
2017-06-14 23:15:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:37 PDT INFO remoteWorkerThread_1: SYNC 5000000154 done in 0.005 seconds
2017-06-14 23:15:37 PDT INFO remoteWorkerThread_3: SYNC 5000000092 done in 0.006 seconds
2017-06-14 23:15:47 PDT INFO remoteWorkerThread_3: SYNC 5000000093 done in 0.005 seconds
2017-06-14 23:15:51 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:51 PDT INFO remoteWorkerThread_1: SYNC 5000000155 done in 0.004 seconds
2017-06-14 23:15:55 PDT INFO remoteWorkerThread_3: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:15:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:59 PDT INFO remoteWorkerThread_1: SYNC 5000000156 done in 0.005 seconds
2017-06-14 23:16:09 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:09 PDT INFO remoteWorkerThread_1: SYNC 5000000157 done in 0.004 seconds
2017-06-14 23:16:12 PDT INFO remoteWorkerThread_3: SYNC 5000000095 done in 0.009 seconds
2017-06-14 23:16:13 PDT INFO remoteWorkerThread_3: SYNC 5000000096 done in 0.015 seconds
2017-06-14 23:16:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:17 PDT INFO remoteWorkerThread_1: SYNC 5000000158 done in 0.004 seconds
2017-06-14 23:16:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:28 PDT INFO remoteWorkerThread_1: SYNC 5000000159 done in 0.004 seconds
2017-06-14 23:16:31 PDT INFO remoteWorkerThread_3: SYNC 5000000097 done in 0.005 seconds
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000160 done in 0.004 seconds
2017-06-14 23:16:39 PDT INFO remoteWorkerThread_3: SYNC 5000000098 done in 0.005 seconds
2017-06-14 23:16:46 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:46 PDT INFO remoteWorkerThread_1: SYNC 5000000161 done in 0.004 seconds
2017-06-14 23:16:50 PDT INFO remoteWorkerThread_3: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:16:57 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:57 PDT INFO remoteWorkerThread_1: SYNC 5000000162 done in 0.004 seconds
2017-06-14 23:17:01 PDT INFO remoteWorkerThread_3: SYNC 5000000100 done in 0.008 seconds
2017-06-14 23:17:09 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:09 PDT INFO remoteWorkerThread_3: SYNC 5000000101 done in 0.006 seconds
2017-06-14 23:17:09 PDT INFO remoteWorkerThread_1: SYNC 5000000163 done in 0.005 seconds
2017-06-14 23:17:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:22 PDT INFO remoteWorkerThread_1: SYNC 5000000164 done in 0.005 seconds
2017-06-14 23:17:22 PDT INFO remoteWorkerThread_3: SYNC 5000000102 done in 0.005 seconds
2017-06-14 23:17:25 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:25 PDT INFO remoteWorkerThread_1: SYNC 5000000165 done in 0.004 seconds
2017-06-14 23:17:30 PDT INFO remoteWorkerThread_3: SYNC 5000000103 done in 0.005 seconds
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: SYNC 5000000166 done in 0.004 seconds
2017-06-14 23:17:40 PDT INFO remoteWorkerThread_3: SYNC 5000000104 done in 0.013 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15559
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15656
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15675
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15683
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15825
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15838
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15853
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15939
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15940
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15941
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1
CONTEXT: PL/pgSQL function _region_il.cleanupevent(interval) line 95 at assignment
2017-06-14 23:17:42 PDT INFO cleanupThread: 0.047 seconds for cleanupEvent()
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_3: SYNC 5000000105 done in 0.005 seconds
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: SYNC 5000000167 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_3: SYNC 5000000106 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_1: SYNC 5000000168 done in 0.005 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_3: SYNC 5000000107 done in 0.006 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: SYNC 5000000169 done in 0.006 seconds
2017-06-14 23:18:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:17 PDT INFO remoteWorkerThread_1: SYNC 5000000170 done in 0.005 seconds
2017-06-14 23:18:20 PDT INFO remoteWorkerThread_3: SYNC 5000000108 done in 0.005 seconds
2017-06-14 23:18:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:28 PDT INFO remoteWorkerThread_3: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:18:28 PDT INFO remoteWorkerThread_1: SYNC 5000000171 done in 0.005 seconds
2017-06-14 23:18:41 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:41 PDT INFO remoteWorkerThread_1: SYNC 5000000172 done in 0.005 seconds
2017-06-14 23:18:41 PDT INFO remoteWorkerThread_3: SYNC 5000000110 done in 0.005 seconds
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_3: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: SYNC 5000000173 done in 0.005 seconds
2017-06-14 23:18:57 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:57 PDT INFO remoteWorkerThread_1: SYNC 5000000174 done in 0.013 seconds
2017-06-14 23:19:00 PDT INFO remoteWorkerThread_3: SYNC 5000000112 done in 0.009 seconds
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_3: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_1: SYNC 5000000175 done in 0.005 seconds
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_3: SYNC 5000000114 done in 0.006 seconds
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_1: SYNC 5000000176 done in 0.005 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_3: SYNC 5000000115 done in 0.006 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_1: SYNC 5000000177 done in 0.006 seconds
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: SYNC 5000000178 done in 0.005 seconds
2017-06-14 23:19:40 PDT INFO remoteWorkerThread_3: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_3: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_1: SYNC 5000000179 done in 0.005 seconds
2017-06-14 23:20:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:01 PDT INFO remoteWorkerThread_1: SYNC 5000000180 done in 0.006 seconds
2017-06-14 23:20:01 PDT INFO remoteWorkerThread_3: SYNC 5000000118 done in 0.006 seconds
2017-06-14 23:20:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:14 PDT INFO remoteWorkerThread_1: SYNC 5000000181 done in 0.005 seconds
2017-06-14 23:20:14 PDT INFO remoteWorkerThread_3: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:20:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:17 PDT INFO remoteWorkerThread_1: SYNC 5000000182 done in 0.005 seconds
2017-06-14 23:20:18 PDT INFO remoteWorkerThread_3: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:20:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:27 PDT INFO remoteWorkerThread_1: SYNC 5000000183 done in 0.005 seconds
2017-06-14 23:20:32 PDT INFO remoteWorkerThread_3: SYNC 5000000121 done in 0.006 seconds
2017-06-14 23:20:38 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:38 PDT INFO remoteWorkerThread_1: SYNC 5000000184 done in 0.004 seconds
2017-06-14 23:20:39 PDT INFO remoteWorkerThread_3: SYNC 5000000122 done in 0.005 seconds
2017-06-14 23:20:53 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:53 PDT INFO remoteWorkerThread_3: SYNC 5000000123 done in 0.006 seconds
2017-06-14 23:20:53 PDT INFO remoteWorkerThread_1: SYNC 5000000185 done in 0.006 seconds
2017-06-14 23:21:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:01 PDT INFO remoteWorkerThread_3: SYNC 5000000124 done in 0.006 seconds
2017-06-14 23:21:01 PDT INFO remoteWorkerThread_1: SYNC 5000000186 done in 0.006 seconds
2017-06-14 23:21:09 PDT INFO remoteWorkerThread_3: SYNC 5000000125 done in 0.005 seconds
2017-06-14 23:21:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:14 PDT INFO remoteWorkerThread_1: SYNC 5000000187 done in 0.004 seconds
2017-06-14 23:21:15 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:15 PDT INFO remoteWorkerThread_1: SYNC 5000000188 done in 0.004 seconds
2017-06-14 23:21:20 PDT INFO remoteWorkerThread_3: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:21:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:26 PDT INFO remoteWorkerThread_1: SYNC 5000000189 done in 0.004 seconds
2017-06-14 23:21:31 PDT INFO remoteWorkerThread_3: SYNC 5000000127 done in 0.005 seconds
2017-06-14 23:21:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:37 PDT INFO remoteWorkerThread_1: SYNC 5000000190 done in 0.005 seconds
2017-06-14 23:21:42 PDT INFO remoteWorkerThread_3: SYNC 5000000128 done in 0.005 seconds
2017-06-14 23:21:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:48 PDT INFO remoteWorkerThread_1: SYNC 5000000191 done in 0.008 seconds
2017-06-14 23:21:52 PDT INFO remoteWorkerThread_3: SYNC 5000000129 done in 0.006 seconds
2017-06-14 23:22:00 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:00 PDT INFO remoteWorkerThread_1: SYNC 5000000192 done in 0.005 seconds
2017-06-14 23:22:00 PDT INFO remoteWorkerThread_3: SYNC 5000000130 done in 0.006 seconds
2017-06-14 23:22:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:13 PDT INFO remoteWorkerThread_3: SYNC 5000000131 done in 0.005 seconds
2017-06-14 23:22:13 PDT INFO remoteWorkerThread_1: SYNC 5000000193 done in 0.005 seconds
2017-06-14 23:22:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:16 PDT INFO remoteWorkerThread_1: SYNC 5000000194 done in 0.004 seconds
2017-06-14 23:22:21 PDT INFO remoteWorkerThread_3: SYNC 5000000132 done in 0.005 seconds
2017-06-14 23:22:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:27 PDT INFO remoteWorkerThread_1: SYNC 5000000195 done in 0.005 seconds
2017-06-14 23:22:32 PDT INFO remoteWorkerThread_3: SYNC 5000000133 done in 0.005 seconds
2017-06-14 23:22:38 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:38 PDT INFO remoteWorkerThread_1: SYNC 5000000196 done in 0.006 seconds
2017-06-14 23:22:42 PDT INFO remoteWorkerThread_3: SYNC 5000000134 done in 0.005 seconds
2017-06-14 23:22:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:50 PDT INFO remoteWorkerThread_1: SYNC 5000000197 done in 0.005 seconds
2017-06-14 23:22:54 PDT INFO remoteWorkerThread_3: SYNC 5000000135 done in 0.005 seconds
2017-06-14 23:22:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:55 PDT INFO remoteWorkerThread_1: SYNC 5000000198 done in 0.004 seconds
2017-06-14 23:22:59 PDT INFO remoteWorkerThread_3: SYNC 5000000136 done in 0.005 seconds
2017-06-14 23:23:10 PDT INFO remoteWorkerThread_3: SYNC 5000000137 done in 0.005 seconds
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: SYNC 5000000199 done in 0.005 seconds
2017-06-14 23:23:21 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:21 PDT INFO remoteWorkerThread_1: SYNC 5000000200 done in 0.004 seconds
2017-06-14 23:23:25 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:25 PDT INFO remoteWorkerThread_3: SYNC 5000000138 done in 0.005 seconds
2017-06-14 23:23:25 PDT INFO remoteWorkerThread_1: SYNC 5000000201 done in 0.005 seconds
2017-06-14 23:23:34 PDT INFO remoteWorkerThread_3: SYNC 5000000139 done in 0.005 seconds
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: SYNC 5000000202 done in 0.005 seconds
2017-06-14 23:23:43 PDT INFO remoteWorkerThread_3: SYNC 5000000140 done in 0.005 seconds
2017-06-14 23:23:47 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:47 PDT INFO remoteWorkerThread_1: SYNC 5000000203 done in 0.004 seconds

辅节点2查看Slony-I日志信息

$ cat /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-14.log
2017-06-14 23:00:38 PDT CONFIG main: slon version 2.2.4 starting up
2017-06-14 23:00:38 PDT INFO slon: watchdog process started
2017-06-14 23:00:38 PDT CONFIG slon: watchdog ready - pid = 19212
2017-06-14 23:00:38 PDT CONFIG slon: worker process created - pid = 19213
2017-06-14 23:00:38 PDT CONFIG main: Integer option vac_frequency = 3
2017-06-14 23:00:38 PDT CONFIG main: Integer option log_level = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_interval = 1000
2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_interval_timeout = 10000
2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_group_maxsize = 20
2017-06-14 23:00:38 PDT CONFIG main: Integer option syslog = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option quit_sync_provider = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option remote_listen_timeout = 300
2017-06-14 23:00:38 PDT CONFIG main: Integer option monitor_interval = 500
2017-06-14 23:00:38 PDT CONFIG main: Integer option explain_interval = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_idle = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_interval = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_count = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option apply_cache_size = 100
2017-06-14 23:00:38 PDT CONFIG main: Boolean option log_pid = 0
2017-06-14 23:00:38 PDT CONFIG main: Boolean option log_timestamp = 1
2017-06-14 23:00:38 PDT CONFIG main: Boolean option tcp_keepalive = 1
2017-06-14 23:00:38 PDT CONFIG main: Boolean option monitor_threads = 1
2017-06-14 23:00:38 PDT CONFIG main: Real option real_placeholder = 0.000000
2017-06-14 23:00:38 PDT CONFIG main: String option cluster_name = region_il
2017-06-14 23:00:38 PDT CONFIG main: String option conn_info = host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
2017-06-14 23:00:38 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node3.pid
2017-06-14 23:00:38 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2017-06-14 23:00:38 PDT CONFIG main: String option archive_dir = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option sql_on_connection = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option lag_interval = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option command_on_logarchive = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option syslog_facility = LOCAL0
2017-06-14 23:00:38 PDT CONFIG main: String option syslog_ident = slon
2017-06-14 23:00:38 PDT CONFIG main: String option cleanup_interval = 10 minutes
2017-06-14 23:00:38 PDT CONFIG main: local node id = 3
2017-06-14 23:00:38 PDT INFO main: main process started
2017-06-14 23:00:38 PDT CONFIG main: launching sched_start_mainloop
2017-06-14 23:00:38 PDT CONFIG main: loading current cluster configuration
2017-06-14 23:00:38 PDT CONFIG storeNode: no_id=1 no_comment='Node 1 - region_il@10.189.102.118'
2017-06-14 23:00:38 PDT CONFIG storeNode: no_id=2 no_comment='Node 2 - region_il@10.189.100.195'
2017-06-14 23:00:38 PDT CONFIG storePath: pa_server=1 pa_client=3 pa_conninfo="host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 23:00:38 PDT CONFIG storePath: pa_server=2 pa_client=3 pa_conninfo="host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG main: last local event sequence = 5000000002
2017-06-14 23:00:38 PDT CONFIG main: configuration complete - starting threads
2017-06-14 23:00:38 PDT INFO localListenThread: thread starts
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG enableNode: no_id=1
2017-06-14 23:00:38 PDT CONFIG enableNode: no_id=2
2017-06-14 23:00:38 PDT INFO main: running scheduler mainloop
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: thread starts
2017-06-14 23:00:38 PDT INFO remoteListenThread_2: thread starts
2017-06-14 23:00:38 PDT CONFIG cleanupThread: thread starts
2017-06-14 23:00:38 PDT INFO syncThread: thread starts
2017-06-14 23:00:38 PDT INFO monitorThread: thread starts
2017-06-14 23:00:38 PDT INFO remoteListenThread_1: thread starts
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: thread starts
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG cleanupThread: bias = 60
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000003 done in 0.019 seconds
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000005 done in 0.011 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000008 done in 0.011 seconds
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000010 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000014 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000009 done in 0.011 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000022 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000017 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000038 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000020 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000043 done in 0.002 seconds
2017-06-14 23:00:46 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:46 PDT INFO remoteWorkerThread_2: SYNC 5000000021 done in 0.003 seconds
2017-06-14 23:00:46 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:46 PDT INFO remoteWorkerThread_1: SYNC 5000000044 done in 0.017 seconds
2017-06-14 23:00:57 PDT INFO remoteWorkerThread_2: SYNC 5000000022 done in 0.002 seconds
2017-06-14 23:01:02 PDT INFO remoteWorkerThread_1: SYNC 5000000045 done in 0.005 seconds
2017-06-14 23:01:03 PDT INFO remoteWorkerThread_1: SYNC 5000000046 done in 0.004 seconds
2017-06-14 23:01:08 PDT INFO remoteWorkerThread_2: SYNC 5000000023 done in 0.003 seconds
2017-06-14 23:01:14 PDT INFO remoteWorkerThread_1: SYNC 5000000047 done in 0.004 seconds
2017-06-14 23:01:19 PDT INFO remoteWorkerThread_2: SYNC 5000000024 done in 0.003 seconds
2017-06-14 23:01:23 PDT INFO remoteWorkerThread_2: SYNC 5000000025 done in 0.003 seconds
2017-06-14 23:01:25 PDT INFO remoteWorkerThread_1: SYNC 5000000048 done in 0.004 seconds
2017-06-14 23:01:34 PDT INFO remoteWorkerThread_2: SYNC 5000000026 done in 0.002 seconds
2017-06-14 23:01:38 PDT INFO remoteWorkerThread_1: SYNC 5000000049 done in 0.004 seconds
2017-06-14 23:01:46 PDT INFO remoteWorkerThread_2: SYNC 5000000027 done in 0.003 seconds
2017-06-14 23:01:50 PDT INFO remoteWorkerThread_1: SYNC 5000000050 done in 0.005 seconds
2017-06-14 23:01:54 PDT INFO remoteWorkerThread_2: SYNC 5000000028 done in 0.002 seconds
2017-06-14 23:01:54 PDT INFO remoteWorkerThread_1: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:02:05 PDT INFO remoteWorkerThread_2: SYNC 5000000029 done in 0.003 seconds
2017-06-14 23:02:09 PDT INFO remoteWorkerThread_1: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:02:17 PDT INFO remoteWorkerThread_1: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:02:21 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:02:21 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:02:21 PDT INFO remoteWorkerThread_2: SYNC 5000000030 done in 0.027 seconds
2017-06-14 23:02:25 PDT INFO remoteWorkerThread_1: SYNC 5000000054 done in 0.007 seconds
2017-06-14 23:02:25 PDT INFO remoteWorkerThread_2: SYNC 5000000031 done in 0.008 seconds
2017-06-14 23:02:36 PDT INFO remoteWorkerThread_1: SYNC 5000000055 done in 0.008 seconds
2017-06-14 23:02:39 PDT INFO remoteWorkerThread_2: SYNC 5000000032 done in 0.005 seconds
2017-06-14 23:02:43 PDT INFO remoteWorkerThread_1: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:02:47 PDT INFO remoteWorkerThread_2: SYNC 5000000033 done in 0.005 seconds
2017-06-14 23:02:54 PDT INFO remoteWorkerThread_1: SYNC 5000000057 done in 0.004 seconds
2017-06-14 23:02:54 PDT INFO remoteWorkerThread_2: SYNC 5000000034 done in 0.005 seconds
2017-06-14 23:03:05 PDT INFO remoteWorkerThread_1: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:03:08 PDT INFO remoteWorkerThread_2: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:03:16 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1 (il_longlive_schema) for region_il'
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_2: SYNC 5000000036 done in 0.005 seconds
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: connection for provider 2 terminated
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 2
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:16 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000060 done in 0.012 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000062 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000064 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000066 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000068 done in 0.003 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000070 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000072 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000074 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000076 done in 0.002 seconds
2017-06-14 23:03:29 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_2: SYNC 5000000037 done in 0.005 seconds
2017-06-14 23:03:29 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_1: SYNC 5000000077 done in 0.015 seconds
2017-06-14 23:03:42 PDT INFO remoteWorkerThread_2: SYNC 5000000038 done in 0.013 seconds
2017-06-14 23:03:42 PDT INFO remoteWorkerThread_1: SYNC 5000000078 done in 0.015 seconds
2017-06-14 23:03:43 PDT INFO remoteWorkerThread_1: SYNC 5000000079 done in 0.004 seconds
2017-06-14 23:03:43 PDT INFO remoteWorkerThread_2: SYNC 5000000039 done in 0.005 seconds
2017-06-14 23:03:56 PDT INFO remoteWorkerThread_1: SYNC 5000000080 done in 0.004 seconds
2017-06-14 23:03:56 PDT INFO remoteWorkerThread_2: SYNC 5000000040 done in 0.005 seconds
2017-06-14 23:04:04 PDT INFO remoteWorkerThread_2: SYNC 5000000041 done in 0.005 seconds
2017-06-14 23:04:08 PDT INFO remoteWorkerThread_1: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:04:16 PDT INFO remoteWorkerThread_1: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:04:16 PDT INFO remoteWorkerThread_2: SYNC 5000000042 done in 0.006 seconds
2017-06-14 23:04:29 PDT INFO remoteWorkerThread_2: SYNC 5000000043 done in 0.004 seconds
2017-06-14 23:04:29 PDT INFO remoteWorkerThread_1: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:04:37 PDT INFO remoteWorkerThread_2: SYNC 5000000044 done in 0.004 seconds
2017-06-14 23:04:37 PDT INFO remoteWorkerThread_1: SYNC 5000000084 done in 0.005 seconds
2017-06-14 23:04:48 PDT INFO remoteWorkerThread_2: SYNC 5000000045 done in 0.004 seconds
2017-06-14 23:04:48 PDT INFO remoteWorkerThread_1: SYNC 5000000085 done in 0.005 seconds
2017-06-14 23:05:02 PDT INFO remoteWorkerThread_1: SYNC 5000000086 done in 0.004 seconds
2017-06-14 23:05:02 PDT INFO remoteWorkerThread_2: SYNC 5000000046 done in 0.005 seconds
2017-06-14 23:05:03 PDT INFO remoteWorkerThread_1: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:05:05 PDT INFO remoteWorkerThread_2: SYNC 5000000047 done in 0.005 seconds
2017-06-14 23:05:14 PDT INFO remoteWorkerThread_1: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:05:19 PDT INFO remoteWorkerThread_2: SYNC 5000000048 done in 0.004 seconds
2017-06-14 23:05:23 PDT INFO remoteWorkerThread_2: SYNC 5000000049 done in 0.005 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_1: SYNC 5000000089 done in 0.004 seconds
2017-06-14 23:05:34 PDT INFO remoteWorkerThread_2: SYNC 5000000050 done in 0.006 seconds
2017-06-14 23:05:38 PDT INFO remoteWorkerThread_1: SYNC 5000000090 done in 0.005 seconds
2017-06-14 23:05:46 PDT INFO remoteWorkerThread_1: SYNC 5000000091 done in 0.004 seconds
2017-06-14 23:05:46 PDT INFO remoteWorkerThread_2: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:05:59 PDT INFO remoteWorkerThread_1: SYNC 5000000092 done in 0.004 seconds
2017-06-14 23:05:59 PDT INFO remoteWorkerThread_2: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:06:12 PDT INFO remoteWorkerThread_2: SYNC 5000000053 done in 0.004 seconds
2017-06-14 23:06:12 PDT INFO remoteWorkerThread_1: SYNC 5000000093 done in 0.004 seconds
2017-06-14 23:06:15 PDT INFO remoteWorkerThread_2: SYNC 5000000054 done in 0.004 seconds
2017-06-14 23:06:15 PDT INFO remoteWorkerThread_1: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:06:28 PDT INFO remoteWorkerThread_2: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:06:28 PDT INFO remoteWorkerThread_1: SYNC 5000000095 done in 0.005 seconds
2017-06-14 23:06:36 PDT INFO remoteWorkerThread_1: SYNC 5000000096 done in 0.004 seconds
2017-06-14 23:06:36 PDT INFO remoteWorkerThread_2: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:06:47 PDT INFO remoteWorkerThread_1: SYNC 5000000097 done in 0.005 seconds
2017-06-14 23:06:48 PDT INFO remoteWorkerThread_2: SYNC 5000000057 done in 0.004 seconds
2017-06-14 23:06:58 PDT INFO remoteWorkerThread_1: SYNC 5000000098 done in 0.004 seconds
2017-06-14 23:06:58 PDT INFO remoteWorkerThread_2: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:07:12 PDT INFO remoteWorkerThread_1: SYNC 5000000099 done in 0.004 seconds
2017-06-14 23:07:12 PDT INFO remoteWorkerThread_2: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:07:15 PDT INFO remoteWorkerThread_2: SYNC 5000000060 done in 0.004 seconds
2017-06-14 23:07:15 PDT INFO remoteWorkerThread_1: SYNC 5000000100 done in 0.005 seconds
2017-06-14 23:07:28 PDT INFO remoteWorkerThread_1: SYNC 5000000101 done in 0.060 seconds
2017-06-14 23:07:28 PDT INFO remoteWorkerThread_2: SYNC 5000000061 done in 0.062 seconds
2017-06-14 23:07:36 PDT INFO remoteWorkerThread_2: SYNC 5000000062 done in 0.004 seconds
2017-06-14 23:07:36 PDT INFO remoteWorkerThread_1: SYNC 5000000102 done in 0.005 seconds
2017-06-14 23:07:47 PDT INFO remoteWorkerThread_1: SYNC 5000000103 done in 0.008 seconds
2017-06-14 23:07:47 PDT INFO remoteWorkerThread_2: SYNC 5000000063 done in 0.009 seconds
2017-06-14 23:07:58 PDT INFO remoteWorkerThread_1: SYNC 5000000104 done in 0.005 seconds
2017-06-14 23:07:58 PDT INFO remoteWorkerThread_2: SYNC 5000000064 done in 0.005 seconds
2017-06-14 23:08:09 PDT INFO remoteWorkerThread_1: SYNC 5000000105 done in 0.007 seconds
2017-06-14 23:08:09 PDT INFO remoteWorkerThread_2: SYNC 5000000065 done in 0.009 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_1: SYNC 5000000106 done in 0.005 seconds
2017-06-14 23:08:19 PDT INFO remoteWorkerThread_2: SYNC 5000000066 done in 0.010 seconds
2017-06-14 23:08:23 PDT INFO remoteWorkerThread_2: SYNC 5000000067 done in 0.004 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_1: SYNC 5000000107 done in 0.008 seconds
2017-06-14 23:08:37 PDT INFO remoteWorkerThread_2: SYNC 5000000068 done in 0.006 seconds
2017-06-14 23:08:41 PDT INFO remoteWorkerThread_1: SYNC 5000000108 done in 0.005 seconds
2017-06-14 23:08:45 PDT INFO remoteWorkerThread_1: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:08:49 PDT INFO remoteWorkerThread_2: SYNC 5000000069 done in 0.005 seconds
2017-06-14 23:08:57 PDT INFO remoteWorkerThread_1: SYNC 5000000110 done in 0.006 seconds
2017-06-14 23:08:57 PDT INFO remoteWorkerThread_2: SYNC 5000000070 done in 0.006 seconds
2017-06-14 23:09:08 PDT INFO remoteWorkerThread_2: SYNC 5000000071 done in 0.005 seconds
2017-06-14 23:09:08 PDT INFO remoteWorkerThread_1: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:09:19 PDT INFO remoteWorkerThread_1: SYNC 5000000112 done in 0.004 seconds
2017-06-14 23:09:19 PDT INFO remoteWorkerThread_2: SYNC 5000000072 done in 0.005 seconds
2017-06-14 23:09:29 PDT INFO remoteWorkerThread_2: SYNC 5000000073 done in 0.004 seconds
2017-06-14 23:09:29 PDT INFO remoteWorkerThread_1: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:09:33 PDT INFO remoteWorkerThread_2: SYNC 5000000074 done in 0.005 seconds
2017-06-14 23:09:34 PDT INFO remoteWorkerThread_1: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:09:43 PDT INFO remoteWorkerThread_2: SYNC 5000000075 done in 0.005 seconds
2017-06-14 23:09:47 PDT INFO remoteWorkerThread_1: SYNC 5000000115 done in 0.004 seconds
2017-06-14 23:09:55 PDT INFO remoteWorkerThread_2: SYNC 5000000076 done in 0.005 seconds
2017-06-14 23:10:01 PDT INFO remoteWorkerThread_1: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:10:05 PDT INFO remoteWorkerThread_1: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:10:06 PDT INFO remoteWorkerThread_2: SYNC 5000000077 done in 0.005 seconds
2017-06-14 23:10:14 PDT INFO remoteWorkerThread_1: SYNC 5000000118 done in 0.005 seconds
2017-06-14 23:10:18 PDT INFO remoteWorkerThread_2: SYNC 5000000078 done in 0.005 seconds
2017-06-14 23:10:26 PDT INFO remoteWorkerThread_2: SYNC 5000000079 done in 0.005 seconds
2017-06-14 23:10:30 PDT INFO remoteWorkerThread_1: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:10:34 PDT INFO remoteWorkerThread_2: SYNC 5000000080 done in 0.005 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19256
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: Logswitch to sl_log_2 initiated
CONTEXT: SQL statement "SELECT "_region_il".logswitch_start()"
PL/pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM
2017-06-14 23:10:39 PDT INFO cleanupThread: 0.021 seconds for cleanupEvent()
2017-06-14 23:10:39 PDT INFO remoteWorkerThread_1: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:10:45 PDT INFO remoteWorkerThread_1: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:10:45 PDT INFO remoteWorkerThread_2: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:10:56 PDT INFO remoteWorkerThread_2: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:10:57 PDT INFO remoteWorkerThread_1: SYNC 5000000122 done in 0.004 seconds
2017-06-14 23:11:07 PDT INFO remoteWorkerThread_1: SYNC 5000000123 done in 0.005 seconds
2017-06-14 23:11:08 PDT INFO remoteWorkerThread_2: SYNC 5000000083 done in 0.004 seconds
2017-06-14 23:11:18 PDT INFO remoteWorkerThread_2: SYNC 5000000084 done in 0.005 seconds
2017-06-14 23:11:19 PDT INFO remoteWorkerThread_1: SYNC 5000000124 done in 0.004 seconds
2017-06-14 23:11:23 PDT INFO remoteWorkerThread_2: SYNC 5000000085 done in 0.004 seconds
2017-06-14 23:11:29 PDT INFO remoteWorkerThread_1: SYNC 5000000125 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_1: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_2: SYNC 5000000086 done in 0.005 seconds
2017-06-14 23:11:45 PDT INFO remoteWorkerThread_1: SYNC 5000000127 done in 0.004 seconds
2017-06-14 23:11:45 PDT INFO remoteWorkerThread_2: SYNC 5000000087 done in 0.004 seconds
2017-06-14 23:11:56 PDT INFO remoteWorkerThread_2: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 2 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 2
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:12:00 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:00 PDT INFO remoteWorkerThread_1: SYNC 5000000130 done in 0.012 seconds
2017-06-14 23:12:08 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:08 PDT INFO remoteWorkerThread_1: SYNC 5000000131 done in 0.003 seconds
2017-06-14 23:12:08 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:08 PDT INFO remoteWorkerThread_2: SYNC 5000000089 done in 0.014 seconds
2017-06-14 23:12:19 PDT INFO remoteWorkerThread_1: SYNC 5000000132 done in 0.002 seconds
2017-06-14 23:12:30 PDT INFO remoteWorkerThread_1: SYNC 5000000133 done in 0.003 seconds
2017-06-14 23:12:34 PDT INFO remoteWorkerThread_1: SYNC 5000000134 done in 0.003 seconds
2017-06-14 23:12:34 PDT INFO remoteWorkerThread_2: SYNC 5000000090 done in 0.003 seconds
2017-06-14 23:12:38 PDT INFO remoteWorkerThread_2: SYNC 5000000091 done in 0.003 seconds
2017-06-14 23:12:49 PDT INFO remoteWorkerThread_1: SYNC 5000000135 done in 0.002 seconds
2017-06-14 23:12:53 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:53 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:56 PDT INFO remoteWorkerThread_1: SYNC 5000000136 done in 0.002 seconds
2017-06-14 23:13:02 PDT INFO remoteWorkerThread_2: SYNC 5000000092 done in 8.409 seconds
2017-06-14 23:13:07 PDT INFO remoteWorkerThread_1: SYNC 5000000137 done in 0.003 seconds
2017-06-14 23:13:07 PDT INFO remoteWorkerThread_2: SYNC 5000000093 done in 0.005 seconds
2017-06-14 23:13:18 PDT INFO remoteWorkerThread_1: SYNC 5000000138 done in 0.003 seconds
2017-06-14 23:13:18 PDT INFO remoteWorkerThread_2: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:13:29 PDT INFO remoteWorkerThread_1: SYNC 5000000139 done in 0.003 seconds
2017-06-14 23:13:29 PDT INFO remoteWorkerThread_2: SYNC 5000000095 done in 0.005 seconds
2017-06-14 23:13:40 PDT INFO remoteWorkerThread_1: SYNC 5000000140 done in 0.003 seconds
2017-06-14 23:13:40 PDT INFO remoteWorkerThread_2: SYNC 5000000096 done in 0.006 seconds
2017-06-14 23:13:44 PDT INFO remoteWorkerThread_1: SYNC 5000000141 done in 0.003 seconds
2017-06-14 23:13:48 PDT INFO remoteWorkerThread_2: SYNC 5000000097 done in 0.006 seconds
2017-06-14 23:13:58 PDT INFO remoteWorkerThread_2: SYNC 5000000098 done in 0.005 seconds
2017-06-14 23:14:02 PDT INFO remoteWorkerThread_1: SYNC 5000000142 done in 0.003 seconds
2017-06-14 23:14:06 PDT INFO remoteWorkerThread_1: SYNC 5000000143 done in 0.002 seconds
2017-06-14 23:14:16 PDT INFO remoteWorkerThread_2: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:14:20 PDT INFO remoteWorkerThread_2: SYNC 5000000100 done in 0.007 seconds
2017-06-14 23:14:20 PDT INFO remoteWorkerThread_1: SYNC 5000000144 done in 0.003 seconds
2017-06-14 23:14:28 PDT INFO remoteWorkerThread_1: SYNC 5000000145 done in 0.003 seconds
2017-06-14 23:14:34 PDT INFO remoteWorkerThread_2: SYNC 5000000101 done in 0.015 seconds
2017-06-14 23:14:39 PDT INFO remoteWorkerThread_1: SYNC 5000000146 done in 0.003 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000147 done in 0.003 seconds
2017-06-14 23:14:51 PDT CONFIG storeSubscribe: sub_set=1 sub_provider=2 sub_forward='t'
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:51 PDT WARN remoteWorkerThread_1: copy set: data provider 2 only on sync 5000000128 - sleep 5 seconds
2017-06-14 23:14:55 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:55 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:55 PDT INFO remoteWorkerThread_2: SYNC 5000000102 done in 0.012 seconds
2017-06-14 23:14:56 PDT INFO copy_set 1 - omit=f - bool=0
2017-06-14 23:14:56 PDT INFO omit is FALSE
2017-06-14 23:14:56 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: connected to provider DB
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity_result"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result_logbook"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: all tables for set 1 found on subscriber
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_entity_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_result_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_logbook_record_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity"
NOTICE: truncate of "il_longlive"."entity" succeeded
2017-06-14 23:15:03 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:15:03 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:03 PDT INFO remoteWorkerThread_2: SYNC 5000000103 done in 0.033 seconds
2017-06-14 23:15:17 PDT CONFIG remoteWorkerThread_1: 125045870 bytes copied for table "il_longlive"."entity"
2017-06-14 23:15:19 PDT INFO remoteWorkerThread_2: SYNC 5000000104 done in 0.006 seconds
2017-06-14 23:15:22 PDT INFO remoteWorkerThread_2: SYNC 5000000105 done in 0.005 seconds
2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: 28.669 seconds to copy table "il_longlive"."entity"
2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity_result"
2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity_result"
NOTICE: truncate of "il_longlive"."entity_result" succeeded
2017-06-14 23:15:34 PDT INFO remoteWorkerThread_2: SYNC 5000000106 done in 0.004 seconds
2017-06-14 23:15:37 PDT CONFIG remoteWorkerThread_1: 66996342 bytes copied for table "il_longlive"."entity_result"
2017-06-14 23:15:45 PDT INFO remoteWorkerThread_2: SYNC 5000000107 done in 0.012 seconds
2017-06-14 23:15:56 PDT INFO remoteWorkerThread_2: SYNC 5000000108 done in 0.004 seconds
2017-06-14 23:16:07 PDT INFO remoteWorkerThread_2: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: 45.106 seconds to copy table "il_longlive"."entity_result"
2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result"
2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result"
NOTICE: truncate of "il_longlive"."result" succeeded
2017-06-14 23:16:11 PDT INFO remoteWorkerThread_2: SYNC 5000000110 done in 0.015 seconds
2017-06-14 23:16:27 PDT CONFIG remoteWorkerThread_1: 105961503 bytes copied for table "il_longlive"."result"
2017-06-14 23:16:29 PDT INFO remoteWorkerThread_2: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:16:32 PDT INFO remoteWorkerThread_2: SYNC 5000000112 done in 0.005 seconds
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 25.058 seconds to copy table "il_longlive"."result"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result_logbook"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result_logbook"
NOTICE: truncate of "il_longlive"."result_logbook" succeeded
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 79 bytes copied for table "il_longlive"."result_logbook"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 0.079 seconds to copy table "il_longlive"."result_logbook"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 101 ("il_longlive"."entity_entity_id_seq") to 2528612
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 103 ("il_longlive"."result_id_seq") to 2611527
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 102 ("il_longlive"."entity_result_id_seq") to 2611527
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 104 ("il_longlive"."result_logbook_record_id_seq") to 1
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: 0.002 seconds to copy sequences
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: copy_set SYNC found, use event seqno 5000000147.
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: 0.015 seconds to build initial setsync status
2017-06-14 23:16:35 PDT INFO copy_set 1 done in 98.961 seconds
2017-06-14 23:16:35 PDT CONFIG enableSubscription: sub_set=1
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1
2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 2
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 2
2017-06-14 23:16:35 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:35 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000151 done in 0.042 seconds
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000155 done in 0.006 seconds
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000159 done in 0.005 seconds
2017-06-14 23:16:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:37 PDT INFO remoteWorkerThread_1: SYNC 5000000160 done in 0.008 seconds
2017-06-14 23:16:41 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:41 PDT INFO remoteWorkerThread_2: SYNC 5000000113 done in 0.013 seconds
2017-06-14 23:16:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:55 PDT INFO remoteWorkerThread_2: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:16:55 PDT INFO remoteWorkerThread_1: SYNC 5000000161 done in 0.008 seconds
2017-06-14 23:16:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:59 PDT INFO remoteWorkerThread_1: SYNC 5000000162 done in 0.007 seconds
2017-06-14 23:17:03 PDT INFO remoteWorkerThread_2: SYNC 5000000115 done in 0.003 seconds
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_2: SYNC 5000000116 done in 0.004 seconds
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_1: SYNC 5000000163 done in 0.008 seconds
2017-06-14 23:17:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:28 PDT INFO remoteWorkerThread_2: SYNC 5000000117 done in 0.004 seconds
2017-06-14 23:17:28 PDT INFO remoteWorkerThread_1: SYNC 5000000165 done in 0.008 seconds
2017-06-14 23:17:32 PDT INFO remoteWorkerThread_2: SYNC 5000000118 done in 0.003 seconds
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: SYNC 5000000166 done in 0.007 seconds
2017-06-14 23:17:47 PDT INFO remoteWorkerThread_2: SYNC 5000000119 done in 0.002 seconds
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: SYNC 5000000167 done in 0.007 seconds
2017-06-14 23:17:52 PDT INFO remoteWorkerThread_2: SYNC 5000000120 done in 0.003 seconds
2017-06-14 23:18:03 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:03 PDT INFO remoteWorkerThread_2: SYNC 5000000121 done in 0.004 seconds
2017-06-14 23:18:03 PDT INFO remoteWorkerThread_1: SYNC 5000000168 done in 0.008 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_2: SYNC 5000000122 done in 0.004 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: SYNC 5000000169 done in 0.007 seconds
2017-06-14 23:18:18 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:18 PDT INFO remoteWorkerThread_1: SYNC 5000000170 done in 0.022 seconds
2017-06-14 23:18:26 PDT INFO remoteWorkerThread_2: SYNC 5000000123 done in 0.003 seconds
2017-06-14 23:18:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:30 PDT INFO remoteWorkerThread_1: SYNC 5000000171 done in 0.007 seconds
2017-06-14 23:18:31 PDT INFO remoteWorkerThread_2: SYNC 5000000124 done in 0.002 seconds
2017-06-14 23:18:44 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:18:44 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:18:44 PDT INFO remoteWorkerThread_2: SYNC 5000000125 done in 0.015 seconds
2017-06-14 23:18:49 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:49 PDT INFO remoteWorkerThread_1: SYNC 5000000172 done in 0.007 seconds
2017-06-14 23:18:53 PDT INFO remoteWorkerThread_2: SYNC 5000000126 done in 0.004 seconds
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: SYNC 5000000173 done in 0.006 seconds
2017-06-14 23:18:58 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:58 PDT INFO remoteWorkerThread_1: SYNC 5000000174 done in 0.006 seconds
2017-06-14 23:19:02 PDT INFO remoteWorkerThread_2: SYNC 5000000127 done in 0.012 seconds
2017-06-14 23:19:10 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:10 PDT INFO remoteWorkerThread_1: SYNC 5000000175 done in 0.015 seconds
2017-06-14 23:19:11 PDT INFO remoteWorkerThread_2: SYNC 5000000128 done in 0.004 seconds
2017-06-14 23:19:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:22 PDT INFO remoteWorkerThread_2: SYNC 5000000129 done in 0.005 seconds
2017-06-14 23:19:22 PDT INFO remoteWorkerThread_1: SYNC 5000000176 done in 0.007 seconds
2017-06-14 23:19:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:36 PDT INFO remoteWorkerThread_2: SYNC 5000000130 done in 0.006 seconds
2017-06-14 23:19:36 PDT INFO remoteWorkerThread_1: SYNC 5000000177 done in 0.007 seconds
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: SYNC 5000000178 done in 0.006 seconds
2017-06-14 23:19:41 PDT INFO remoteWorkerThread_2: SYNC 5000000131 done in 0.005 seconds
2017-06-14 23:19:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:55 PDT INFO remoteWorkerThread_2: SYNC 5000000132 done in 0.006 seconds
2017-06-14 23:19:55 PDT INFO remoteWorkerThread_1: SYNC 5000000179 done in 0.007 seconds
2017-06-14 23:20:06 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:06 PDT INFO remoteWorkerThread_2: SYNC 5000000133 done in 0.005 seconds
2017-06-14 23:20:06 PDT INFO remoteWorkerThread_1: SYNC 5000000180 done in 0.007 seconds
2017-06-14 23:20:16 PDT INFO remoteWorkerThread_2: SYNC 5000000134 done in 0.007 seconds
2017-06-14 23:20:20 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:20 PDT INFO remoteWorkerThread_1: SYNC 5000000182 done in 0.007 seconds
2017-06-14 23:20:21 PDT INFO remoteWorkerThread_2: SYNC 5000000135 done in 0.005 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_2: SYNC 5000000136 done in 0.006 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_1: SYNC 5000000183 done in 0.008 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19234
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19239
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19244
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19348
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19566
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19567
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19568
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19663
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1
CONTEXT: PL/pgSQL function _region_il.cleanupevent(interval) line 95 at assignment
2017-06-14 23:20:39 PDT INFO cleanupThread: 0.077 seconds for cleanupEvent()
2017-06-14 23:20:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:39 PDT INFO remoteWorkerThread_1: SYNC 5000000184 done in 0.008 seconds
2017-06-14 23:20:43 PDT INFO remoteWorkerThread_2: SYNC 5000000137 done in 0.005 seconds
2017-06-14 23:20:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:54 PDT INFO remoteWorkerThread_2: SYNC 5000000138 done in 0.005 seconds
2017-06-14 23:20:54 PDT INFO remoteWorkerThread_1: SYNC 5000000185 done in 0.007 seconds
2017-06-14 23:21:04 PDT INFO remoteWorkerThread_2: SYNC 5000000139 done in 0.009 seconds
2017-06-14 23:21:08 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:08 PDT INFO remoteWorkerThread_1: SYNC 5000000186 done in 0.006 seconds
2017-06-14 23:21:11 PDT INFO remoteWorkerThread_2: SYNC 5000000140 done in 0.005 seconds
2017-06-14 23:21:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:16 PDT INFO remoteWorkerThread_1: SYNC 5000000188 done in 0.007 seconds
2017-06-14 23:21:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:27 PDT INFO remoteWorkerThread_2: SYNC 5000000141 done in 0.006 seconds
2017-06-14 23:21:27 PDT INFO remoteWorkerThread_1: SYNC 5000000189 done in 0.007 seconds
2017-06-14 23:21:31 PDT INFO remoteWorkerThread_2: SYNC 5000000142 done in 0.005 seconds
2017-06-14 23:21:45 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:45 PDT INFO remoteWorkerThread_2: SYNC 5000000143 done in 0.005 seconds
2017-06-14 23:21:45 PDT INFO remoteWorkerThread_1: SYNC 5000000190 done in 0.007 seconds
2017-06-14 23:21:49 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:49 PDT INFO remoteWorkerThread_1: SYNC 5000000191 done in 0.008 seconds
2017-06-14 23:21:52 PDT INFO remoteWorkerThread_2: SYNC 5000000144 done in 0.005 seconds
2017-06-14 23:22:05 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:05 PDT INFO remoteWorkerThread_2: SYNC 5000000145 done in 0.006 seconds
2017-06-14 23:22:05 PDT INFO remoteWorkerThread_1: SYNC 5000000192 done in 0.008 seconds
2017-06-14 23:22:18 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:18 PDT INFO remoteWorkerThread_2: SYNC 5000000146 done in 0.005 seconds
2017-06-14 23:22:18 PDT INFO remoteWorkerThread_1: SYNC 5000000194 done in 0.007 seconds
2017-06-14 23:22:22 PDT INFO remoteWorkerThread_2: SYNC 5000000147 done in 0.005 seconds
2017-06-14 23:22:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:30 PDT INFO remoteWorkerThread_1: SYNC 5000000195 done in 0.016 seconds
2017-06-14 23:22:31 PDT INFO remoteWorkerThread_2: SYNC 5000000148 done in 0.005 seconds
2017-06-14 23:22:45 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:45 PDT INFO remoteWorkerThread_2: SYNC 5000000149 done in 0.006 seconds
2017-06-14 23:22:45 PDT INFO remoteWorkerThread_1: SYNC 5000000196 done in 0.008 seconds
2017-06-14 23:22:56 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:56 PDT INFO remoteWorkerThread_2: SYNC 5000000150 done in 0.006 seconds
2017-06-14 23:22:56 PDT INFO remoteWorkerThread_1: SYNC 5000000198 done in 0.008 seconds
2017-06-14 23:23:10 PDT INFO remoteWorkerThread_2: SYNC 5000000151 done in 0.005 seconds
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_2: SYNC 5000000152 done in 0.017 seconds
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: SYNC 5000000199 done in 0.019 seconds
2017-06-14 23:23:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:26 PDT INFO remoteWorkerThread_2: SYNC 5000000153 done in 0.005 seconds
2017-06-14 23:23:26 PDT INFO remoteWorkerThread_1: SYNC 5000000201 done in 0.007 seconds
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_2: SYNC 5000000154 done in 0.006 seconds
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: SYNC 5000000202 done in 0.007 seconds
2017-06-14 23:23:43 PDT INFO remoteWorkerThread_2: SYNC 5000000155 done in 0.004 seconds
2017-06-14 23:23:51 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:51 PDT INFO remoteWorkerThread_2: SYNC 5000000156 done in 0.005 seconds
2017-06-14 23:23:51 PDT INFO remoteWorkerThread_1: SYNC 5000000203 done in 0.007 seconds
2017-06-14 23:23:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:59 PDT INFO remoteWorkerThread_1: SYNC 5000000204 done in 0.007 seconds
2017-06-14 23:24:07 PDT INFO remoteWorkerThread_2: SYNC 5000000157 done in 0.005 seconds
2017-06-14 23:24:10 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:10 PDT INFO remoteWorkerThread_1: SYNC 5000000205 done in 0.008 seconds
2017-06-14 23:24:13 PDT INFO remoteWorkerThread_2: SYNC 5000000158 done in 0.005 seconds
2017-06-14 23:24:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:16 PDT INFO remoteWorkerThread_1: SYNC 5000000206 done in 0.006 seconds
2017-06-14 23:24:24 PDT INFO remoteWorkerThread_2: SYNC 5000000159 done in 0.005 seconds
2017-06-14 23:24:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:37 PDT INFO remoteWorkerThread_2: SYNC 5000000160 done in 0.006 seconds
2017-06-14 23:24:37 PDT INFO remoteWorkerThread_1: SYNC 5000000207 done in 0.008 seconds
2017-06-14 23:24:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:50 PDT INFO remoteWorkerThread_2: SYNC 5000000161 done in 0.005 seconds
2017-06-14 23:24:50 PDT INFO remoteWorkerThread_1: SYNC 5000000209 done in 0.007 seconds
2017-06-14 23:24:53 PDT INFO remoteWorkerThread_2: SYNC 5000000162 done in 0.005 seconds
2017-06-14 23:25:06 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:06 PDT INFO remoteWorkerThread_2: SYNC 5000000163 done in 0.013 seconds
2017-06-14 23:25:06 PDT INFO remoteWorkerThread_1: SYNC 5000000210 done in 0.014 seconds
2017-06-14 23:25:19 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:19 PDT INFO remoteWorkerThread_2: SYNC 5000000164 done in 0.005 seconds
2017-06-14 23:25:19 PDT INFO remoteWorkerThread_1: SYNC 5000000211 done in 0.007 seconds
2017-06-14 23:25:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:22 PDT INFO remoteWorkerThread_2: SYNC 5000000165 done in 0.006 seconds
2017-06-14 23:25:22 PDT INFO remoteWorkerThread_1: SYNC 5000000212 done in 0.008 seconds
2017-06-14 23:25:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:30 PDT INFO remoteWorkerThread_1: SYNC 5000000213 done in 0.006 seconds
2017-06-14 23:25:31 PDT INFO remoteWorkerThread_2: SYNC 5000000166 done in 0.004 seconds

13. 查看数据同步状态

$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "\d il_longlive.entity" -W
Password for user slonyadmin:
           Table "il_longlive.entity"
 Column |   Type   |        Modifiers
--------------+------------------------+------------------------------------------------------------------------
 entity_id | integer    | not null default nextval('il_longlive.entity_entity_id_seq'::regclass)
 country_code | character varying(3) |
 state_code | character varying(50) |
 name   | character varying(128) |
 islive  | boolean    |
Indexes:
 "entity_pkey" PRIMARY KEY, btree (entity_id)
Triggers:        #可以看到Slony-I配置完成后原表多了触发器
 _region_il_denyaccess BEFORE INSERT OR DELETE OR UPDATE ON il_longlive.entity FOR EACH ROW EXECUTE PROCEDURE _region_il.denyaccess('_region_il')
 _region_il_truncatedeny BEFORE TRUNCATE ON il_longlive.entity FOR EACH STATEMENT EXECUTE PROCEDURE _region_il.deny_truncate()
Disabled user triggers:
 _region_il_logtrigger AFTER INSERT OR DELETE OR UPDATE ON il_longlive.entity FOR EACH ROW EXECUTE PROCEDURE _region_il.logtrigger('_region_il', '101', 'k')
 _region_il_truncatetrigger BEFORE TRUNCATE ON il_longlive.entity FOR EACH STATEMENT EXECUTE PROCEDURE _region_il.log_truncate('101')
$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W
Password for user slonyadmin:
 count
---------
(1 row)
$ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W
Password for user slonyadmin:
 count
---------
   #可以看到数据已经同步到辅节点1了
(1 row)
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W
Password for user slonyadmin:
 count
---------
  #可以看到数据已经同步到辅节点2了
(1 row)

14. Slony-I相关表或者视图查看

sl_apply_stats: 每个节点本地的复制表的DML统计状态

sl_components: slon/slonik组件等当前的工作状态

region_il=# select * from sl_components;
  co_actor  | co_pid | co_node | co_connection_pid | co_activity |  co_starttime  | co_event | co_eventtype
----------------------+--------+---------+-------------------+------------------+------------------------+------------+--------------
 local_monitor  | 2455 |  0 |    2470 | thread main loop | 2017-06-14 22:54:42-07 |   | n/a
 local_cleanup  | 2455 |  0 |    2468 | cleanupEvent  | 2017-06-15 00:44:48-07 |   | n/a
 remoteWorkerThread_2 | 2455 |  2 |    2471 | SYNC    | 2017-06-15 00:49:14-07 | 5000000668 | SYNC
 local_listen   | 2455 |  1 |    2459 | thread main loop | 2017-06-15 00:49:20-07 |   | n/a
 remoteWorkerThread_3 | 2455 |  3 |    2467 | SYNC    | 2017-06-15 00:49:17-07 | 5000000653 | SYNC
 remote listener  | 2455 |  3 |    19059 | thread main loop | 2017-06-15 00:49:18-07 |   | n/a
 local_sync   | 2455 |  0 |    2469 | thread main loop | 2017-06-15 00:49:21-07 |   | n/a
(7 rows)

sl_table: 当前集群中的表信息

region_il=# select * from sl_table ;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
(4 rows)

sl_set: 当前集群中的集合信息

region_il=# select * from sl_set;
 set_id | set_origin | set_locked |    set_comment
--------+------------+------------+------------------------------------------
  1 |   1 |   | Set 1 (il_longlive_schema) for region_il
(1 row)

sl_set: 当前集群中的节点信息

region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
| t   | Node 3 - region_il@10.189.100.226 | f
(3 rows)

sl_status: 显示远程节点落后主节点的延迟

region_il=# select * from sl_status;
-[ RECORD 1 ]-------------+------------------------------
st_origin     | 1
st_received    | 3
st_last_event    | 5000000800
st_last_event_ts   | 2017-06-15 01:03:19.585161-07
st_last_received   | 5000000799
st_last_received_ts  | 2017-06-15 01:03:14.281766-07
st_last_received_event_ts | 2017-06-15 01:03:09.57357-07
st_lag_num_events   | 1
st_lag_time    | 00:00:11.414753
-[ RECORD 2 ]-------------+------------------------------
st_origin     | 1
st_received    | 2
st_last_event    | 5000000800
st_last_event_ts   | 2017-06-15 01:03:19.585161-07
st_last_received   | 5000000799
st_last_received_ts  | 2017-06-15 01:03:14.193743-07
st_last_received_event_ts | 2017-06-15 01:03:09.57357-07
st_lag_num_events   | 1
st_lag_time    | 00:00:11.414753

sl_subscribe: 显示当前集群中的订阅者

region_il=# select * from sl_subscribe;
 sub_set | sub_provider | sub_receiver | sub_forward | sub_active
---------+--------------+--------------+-------------+------------
|   1 |   2 | t   | t
|   2 |   3 | t   | t
(2 rows)

 Slony-I日常维护

1. 更改主表的表结构结构

1.1. 查看要修改的表结构信息

region_il=# \d result_logbook;
           Table "il_longlive.result_logbook"
  Column  |   Type    |        Modifiers
------------------+-----------------------------+--------------------------------------------------------------------
 record_id  | integer      | not null default nextval('result_logbook_record_id_seq'::regclass)
 schema   | character varying(255)  | not null
 vendor_code  | character varying(255)  |
 tag    | character varying(255)  |
 total_result_cnt | integer      |
 exist_result_cnt | integer      |
 new_result_cnt | integer      |
 max_result_id | integer      |
 islive   | boolean      | default true
 create_time  | timestamp without time zone | default now()
 update_time  | timestamp without time zone |
Indexes:
 "result_logbook_pkey" PRIMARY KEY, btree (record_id)

1.2. 创建修改表字段的ddl语句文件

$ cat /home/postgres/alter_table_result_logbook.sql
alter table il_longlive.result_logbook add column address varchar(40);

1.3. 通过执行slonik_execute_script命令

$ /usr/local/pgsql/bin/slonik_execute_script 1 /home/postgres/alter_table_result_logbook.sql
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 execute script (
 set id = 1,
 filename = '/home/postgres/alter_table_result_logbook.sql',
 event node = 1
 );
$ cat /home/postgres/alter_table_result_logbook.sh
#!/bin/sh
slonik << __EOF__
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 execute script (
 filename = '/home/postgres/alter_table_result_logbook.sql',
 event node = 1
 );
__EOF__
$ chmod +x /home/postgres/alter_table_result_logbook.sh
$ /home/postgres/alter_table_result_logbook.sh

1.4. 验证所有节点上的表结构修改是否成功

$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address
Password for user slonyadmin:
 address   | character varying(40)
$ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address
Password for user slonyadmin:
 address   | character varying(40)
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address
Password for user slonyadmin:
 address   | character varying(40)

2. 向现有集群中增加一个复制表

2.1. 查看主库的的表结构信息,确认符合Slony-I的要求(主键)

region_il=# \d snapshot
          Table "il_longlive.snapshot"
  Column  |    Type    |      Modifiers
-----------------+--------------------------------+-------------------------------------------------------
 id    | integer      | not null default nextval('snapshot_id_seq'::regclass)
 feed_schema  | character varying(255)   |
 job    | character varying(255)   |
 table_name  | character varying(255)   |
 max_primary_key | bigint       |
 create_time  | timestamp(0) without time zone |
 update_time  | timestamp(0) without time zone |
Indexes:
 "snapshot_pkey" PRIMARY KEY, btree (id)

2.2. 将主表结构导入到所有辅节点

$ pg_dump -U slonyadmin -s -t il_longlive.snapshot region_il | psql -h 10.189.100.195 -U slonyadmin region_il
$ pg_dump -U slonyadmin -s -t il_longlive.snapshot region_il | psql -h 10.189.100.226 -U slonyadmin region_il

2.3. 修改Slony-I配置文件slon_tools_conf,增加新复制集,并复制到所有节点(不是必须,为了各节点配置文件完整性)

$ cat /usr/local/pgsql/etc/slon_tools.conf
if ($ENV{"SLONYNODES"}) {
 require $ENV{"SLONYNODES"};
} else {
 $CLUSTER_NAME = 'region_il';
 $PIDFILE_DIR = '/usr/local/pgsql/data';
 $LOGDIR = '/usr/local/pgsql/data/pg_log';
 $MASTERNODE = 1;
 $DEBUGLEVEL = 0;
 $START_WATCHDOG = 1;
 $SLEEP_TIME = 60;
 $WATCHDOG_VERBOSE = 0;

 add_node(node  => 1,
    host  => '10.189.102.118',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);

 add_node(node  => 2,
    parent => 1,
    host  => '10.189.100.195',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);

 add_node(node  => 3,
    parent => 2,
    host  => '10.189.100.226',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);
}

$SLONY_SETS = {
 "il_longlive_schema" => {
  "set_id" => 1,
  "table_id" => 101,
  "sequence_id" => 101,
  "pkeyedtables" => [
       'il_longlive.entity',
       'il_longlive.entity_result',
       'il_longlive.result',
       'il_longlive.result_logbook',
       ],

  "keyedtables" => {},

  "sequences" => ['il_longlive.entity_entity_id_seq',
    'il_longlive.entity_result_id_seq',
   'il_longlive.result_id_seq',
   'il_longlive.result_logbook_record_id_seq',
      ],
 },

 "il_longlive_schema1" => {
  "set_id" => 2,
  "table_id" => 201,
  "sequence_id" => 201,
  "pkeyedtables" => [
       'il_longlive.snapshot',
       ],

  "keyedtables" => {},

  "sequences" => ['il_longlive.snapshot_id_seq'
      ],
 },

};

if ($ENV{"SLONYSET"}) {
 require $ENV{"SLONYSET"};
}

# Please do not add or change anything below this point.
 1;
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.195:/usr/local/pgsql/etc/slon_tools.conf
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.226:/usr/local/pgsql/etc/slon_tools.conf

2.4. 在主节点创建新复制集

$ /usr/local/pgsql/bin/slonik_create_set --config /usr/local/pgsql/etc/slon_tools.conf 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

# TABLE ADD KEY

# CREATE SET
 create set (id = 2, origin = 1, comment = 'Set 2 (il_longlive_schema1) for region_il');

# SET ADD TABLE
 echo 'Subscription set 2 (il_longlive_schema1) created';
 echo 'Adding tables to the subscription set';
 set add table (set id = 2, origin = 1, id = 201,
     full qualified name = 'il_longlive.snapshot',
     comment = 'Table il_longlive.snapshot with primary key');
 echo 'Add primary keyed table il_longlive.snapshot';

# SET ADD SEQUENCE
 echo 'Adding sequences to the subscription set';
 set add sequence (set id = 2, origin = 1, id = 201,
     full qualified name = 'il_longlive.snapshot_id_seq',
     comment = 'Sequence il_longlive.snapshot_id_seq');
 echo 'Add sequence il_longlive.snapshot_id_seq';
 echo 'All tables added';
$ /usr/local/pgsql/bin/slonik_create_set --config /usr/local/pgsql/etc/slon_tools.conf 2 | slonik
<stdin>:12: Subscription set 2 (il_longlive_schema1) created
<stdin>:13: Adding tables to the subscription set
<stdin>:17: Add primary keyed table il_longlive.snapshot
<stdin>:20: Adding sequences to the subscription set
<stdin>:24: Add sequence il_longlive.snapshot_id_seq
<stdin>:25: All tables added

2.5. 在主节点创建新复制集的订阅者

$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

 subscribe set (id = 2, provider = 1, receiver = 2, forward = yes);
 echo 'Subscribed nodes to set 2';
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 3
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

 subscribe set (id = 2, provider = 2, receiver = 3, forward = yes);
 echo 'Subscribed nodes to set 2';
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 2 | slonik
<stdin>:7: Subscribed nodes to set 2
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 3 | slonik
<stdin>:7: Subscribed nodes to set 2

2.6. 查看所有辅节点数据同步状态

$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"  #主节点
Password for user slonyadmin:
 count
-------
(1 row)

$ psql -h 10.189.100.196 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"  #辅节点1
^C
[postgres@ec2t-pgtest-01 bin]$ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"
Password for user slonyadmin:
 count
-------
(1 row)

$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"  #辅节点2
Password for user slonyadmin:
 count
-------
(1 row)

2.7. 查看Slony-I系统表信息

region_il=# select * from sl_set;
 set_id | set_origin | set_locked |    set_comment
--------+------------+------------+-------------------------------------------
|   1 |   | Set 1 (il_longlive_schema) for region_il
|   1 |   | Set 2 (il_longlive_schema1) for region_il
(2 rows)

region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
|  19380 | snapshot  | il_longlive |  2 | snapshot_pkey  | f   | Table il_longlive.snapshot with primary key
(5 rows)

2.8. 合并复制集

$ /usr/local/pgsql/bin/slonik_merge_sets --config /usr/local/pgsql/etc/slon_tools.conf 1 1 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 merge set (id = 1, add id = 2, origin = 1);
 echo 'Replication set 2 merged into 1 on origin 1. Set 2 no longer exists.';
$ /usr/local/pgsql/bin/slonik_merge_sets --config /usr/local/pgsql/etc/slon_tools.conf 1 1 2 | slonik
<stdin>:6: Replication set 2 merged into 1 on origin 1. Set 2 no longer exists.

2.9. 查看Slony-I系统表信息

region_il=# select * from sl_set;
 set_id | set_origin | set_locked |    set_comment
--------+------------+------------+------------------------------------------
|   1 |   | Set 1 (il_longlive_schema) for region_il
(1 row)

region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
|  19380 | snapshot  | il_longlive |  1 | snapshot_pkey  | f   | Table il_longlive.snapshot with primary key
(5 rows)

3. 向现有集群中移除一个复制表

3.1. 查看Slony-I系统视图确认要移除的表

region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
|  19380 | snapshot  | il_longlive |  1 | snapshot_pkey  | f   | Table il_longlive.snapshot with primary key
(5 rows)

3.2. 移除集群中指定的表

$ /usr/local/pgsql/bin/slonik_drop_table --config /usr/local/pgsql/etc/slon_tools.conf 201 1
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

# DROP TABLE
 try {
 SET DROP TABLE (id = 201, origin = 1);
 } on error {
 echo 'Could not drop table 201 for region_il!';
 exit 1;
 }

$ /usr/local/pgsql/bin/slonik_drop_table --config /usr/local/pgsql/etc/slon_tools.conf 201 1 | slonik

3.3. 验证集群中指定的表是否移除

region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
(4 rows)

region_il=# \d il_longlive.snapshot  #可以看到对应的表上的触发器也被移除了
            Table "il_longlive.snapshot"
  Column  |    Type    |        Modifiers
-----------------+--------------------------------+-------------------------------------------------------------------
 id    | integer      | not null default nextval('il_longlive.snapshot_id_seq'::regclass)
 feed_schema  | character varying(255)   |
 job    | character varying(255)   |
 table_name  | character varying(255)   |
 max_primary_key | bigint       |
 create_time  | timestamp(0) without time zone |
 update_time  | timestamp(0) without time zone |
Indexes:
 "snapshot_pkey" PRIMARY KEY, btree (id)

4. 向现有集群中移除一个节点

4.1. 查看Slony-I系统视图确认要移除的节点

region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
| t   | Node 3 - region_il@10.189.100.226 | f
(3 rows)

4.2. 移除集群中指定的节点

$ /usr/local/pgsql/bin/slonik_drop_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 drop node (id = 3, event node = 1);
 echo 'dropped node 3 cluster';
$ /usr/local/pgsql/bin/slonik_drop_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1 | slonik
debug: waiting for 3,5000019408 on 1
debug: waiting for 3,5000019408 on 1
<stdin>:6: dropped node 3 cluster

4.3. 验证集群中指定的节点是否移除(此时被移除节点的Slony-I系统表等信息已经被移除)

region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
(2 rows)

4.4. 停止被关闭节点的slon进程

$ ps -ef | grep slon
postgres 6033  1 0 Jun02 ?  00:00:05 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 6050  1 0 Jun02 ?  00:00:03 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60
postgres 12596 6033 0 05:00 ?  00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 12598 19212 0 05:00 ?  00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 12599 15237 0 05:00 ?  00:00:00 postgres: slonyadmin region_il 10.189.100.226(29630) idle
postgres 12601 5757 0 05:00 pts/3 00:00:00 grep slon
postgres 19212  1 0 Jun14 ?  00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 19232  1 0 Jun14 ?  00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node3 60
$ slon_status 3
Slon is running for the 'region_il' cluster on node3.
$ slon_kill
slon_kill.pl... Killing all slon and slon_watchdog instances for the cluster region_il
1. Kill slon watchdogs
slon_watchdog for cluster region_il node 2 killed - PID [6050]
slon_watchdog for cluster region_il node 3 killed - PID [19232]
$ ps -ef | grep slon
postgres 12820 5757 0 05:05 pts/3 00:00:00 grep slon

5. 向现有集群中增加一个节点

5.1. 在新节点安装配置Slony-I(参考之前步骤)

5.2. 查看当街Slony-I系统表节点信息

region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
(2 rows)

5.3. 向Slony-I配置文件slon_tools.conf 增加新节点信息,并将新配置文件更新到所有节点

$ cat /usr/local/pgsql/etc/slon_tools.conf
...........................................
 add_node(node  => 3,
    parent => 1,
    host  => '10.189.100.226',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);
............................................

5.4. 在主节点上增加向集群新节点信息

$ /usr/local/pgsql/bin/slonik_add_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1

# ADD NODE
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 try {
  store node (id = 3, event node = 1, comment = 'Node 3 - region_il@10.189.100.226');
 } on error {
  echo 'Failed to add node to cluster';
  exit 1;
 }

# STORE PATHS
 store path (server = 3, client = 1, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 1, client = 3, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 3, client = 2, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 2, client = 3, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888');
 echo 'added node 3 to cluster';
 echo 'Please start a slon replication daemon for node 3';
$ /usr/local/pgsql/bin/slonik_add_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1 | slonik
<stdin>:7: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:19: added node 3 to cluster
<stdin>:20: Please start a slon replication daemon for node 3

5.5. 在新节点启动slon进程

$ slon_start 3
Invoke slon for node 3 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-17.log 2>&1 &
Slon successfully started for cluster region_il, node node3
PID [13136]
Start the watchdog process as well...

5.6. 在主节点添加新节点的复制订阅信息

$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/etc/slon_tools.conf 1 3
Can't locate /usr/local/pgsql/etc/etc/slon_tools.conf in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/pgsql/bin/slonik_subscribe_set line 29.
[postgres@ec2t-pgtest-01 bin]$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 1 3
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';

 subscribe set (id = 1, provider = 1, receiver = 3, forward = yes);
 echo 'Subscribed nodes to set 1';
[postgres@ec2t-pgtest-01 bin]$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 1 3 | slonik
<stdin>:7: Subscribed nodes to set 1

5.7. 验证数据同步(略)

总结

以上就是这篇文章的全部内容了,希望本文的内容对大家的学习或者工作具有一定的参考学习价值,如果有疑问大家可以留言交流,谢谢大家对我们的支持。

(0)

相关推荐

  • PostgreSQL 数据库性能提升的几个方面

    1.使用EXPLAIN EXPLAIN命令可以查看执行计划,在前面的blog中已经介绍过.这个方法是我们最主要的调试工具. 2.及时更新执行计划中使用的统计信息 由于统计 信息不是每次操作数据 库 都 进 行更新的,一般是在 VACUUM . ANALYZE . CREATE INDEX等DDL执行的时候会更新统计信息, 因此执 行 计 划所用的 统计 信息很有可能比 较 旧. 这样执 行 计 划的分析 结 果可能 误 差会 变 大. 以下是表tenk1的相关的一部分统计信息. SELECT r

  • Postgresql主从异步流复制方案的深入探究

    前言 数据库的备份工作在日常生产中极为重要,如果你咨询一个DBA如何才能设计出高可用的数据备份与恢复方案,相信很多人都会从架构上给出很多容灾的意见.但归根到底,如果业务环节中数据库还牵涉到分布式环境,我认为一个好的方案需要达到三大要求: 多副本 持久化 一致性 日常架构设计中,我们不仅要保证数据额的成功备份,还要保证备份的数据可以快速恢复.在众多备份恢复可靠性方案中 主从复制 技术,可以说是最常见的实现,本文主要是介绍postgresql主备数据库的异步流复制的环境搭建与主备切换的操作实践,除了

  • PostgreSQL新手入门教程

    自从MySQL被Oracle收购以后,PostgreSQL逐渐成为开源关系型数据库的首选. 本文介绍PostgreSQL的安装和基本用法,供初次使用者上手.以下内容基于Debian操作系统,其他操作系统实在没有精力兼顾,但是大部分内容应该普遍适用. 安装 1.首先,安装PostgreSQL客户端. sudo apt-get install postgresql-client 然后,安装PostgreSQL服务器. sudo apt-get install postgresql 2.正常情况下,安

  • PostgreSQL 安装和简单使用第1/2页

    据我了解国内四大国产数据库,其中三个都是基于PostgreSQL开发的.并且,因为许可证的灵活,任何人都可以以任何目的免费使用,修改,和分发 PostgreSQL,不管是私用,商用,还是学术研究使用.本文只是简单介绍一下postgresql的安装和简单的使用,语法方面涉及的比较少,以方便新手上路为目的. 1.系统环境和安装方法 : PostgreSQL的安装方法比较灵活,可以用源码包安装,也可以用您使用的发行版所带的软件包来安装,还可以采用在线安装-- 1.1 系统环境:Ubuntu Linux

  • 15个postgresql数据库实用命令分享

    最初是想找postgresql数据库占用空间命令发现的这篇blog,发现其中提供的几 条命令很有用(但也有几条感觉是充数的=.=),于是就把它翻译过来了.另外这篇文章是09年的,所以里面的内容可能有点过时,我收集了原文中有用的评论放在了最后面. 现在有不少开源软件都在使用postgreSQL作为它们的数据库系统.但公司可能不会招一些全职的postgreSQL DBA来维护它(piglei: 在国内基本也找不到).而会让一些比如说Oracle DBA.Linux系统管理员或者程序员去 维护.在这篇

  • Windows下Postgresql数据库的下载与配置方法

    注意下载的是二进制版,不是带Windows Installer的. http://www.enterprisedb.com/products-services-training/pgbindownload x86下载http://get.enterprisedb.com/postgresql/postgresql-9.2.4-1-windows-binaries.zip x64下载http://get.enterprisedb.com/postgresql/postgresql-9.2.4-1-

  • Windows下PostgreSQL安装图解

    现在谈起免费数据库,大多数人首先想到的可能是MySQL,的确MySQL目前已经应用在国内很多领域,尤其是网站架设方面.但是,实际上功能最强大.特性最丰富和最复杂的免费数据库应该是PostgreSQL.它的很多特性正是当今许多商业数据库例如Oracle.DB2等的前身. 其实笔者最近也是因为项目需要,接触了一点PostgreSQL的皮毛,最近PostgreSQL又刚发布了8.1版本,笔者结合网上各位高手的经验谈一点自己的安装心得,和才开始接触PostgreSQL的新手朋友共同学习. 从Postgr

  • 使用Bucardo5实现PostgreSQL的主数据库复制

    下一代异步多个主数据库复制系统Bucardo 5发布了.这个版本删除了老版本中两个数据库源的限制,允许有更多的源数据库(即主数据库)以及更多的目标数据库(即备份数据库).Bucardo还可以复制到其他类型的目标数据库,其中包括MySQL.MariaDB.Oracle.SQLite.MongoDB和Redis.Bucardo已经被完全重写了,这个版本比前一版本Bucardo 4功能更强大,效率更高.你可以访问Bucardo wiki查找最新版本的Bucardo. 这篇文章快速的介绍了一下Bucar

  • PostgreSQL中Slony-I同步复制部署教程

    前言 本文主要介绍了关于PostgreSQL中Slony-I同步复制部署的相关内容,分享出来供大家参考学习,下面话不多说了,来一起看看详细的介绍吧 本次测试环境 IP 10.189.102.118 10.189.100.195 10.189.100.226 PGHOME /usr/local/pgsql /usr/local/pgsql /usr/local/pgsql Role Origin,Providers Subscribers,Providers Subscribers Databas

  • Mysql主从三种复制模式(异步复制,半同步复制,组复制)

    目录 MySQL异步复制 半同步复制 组复制 MGR的解决方案现在具备的特性 MGR的解决方案目前的影响 MySQL异步复制 MySQL异步复制是主从复制过程中默认的复制模式.主从复制涉及三个线程,master I/O线程.slave I/O线程.slave sql线程.因为是异步复制,所以master事务的提交,不需要经过slave的确认,即master I/O线程提交事务后,不需要等待slave I/O线程的回复确认,master并不保证binlog一定写入到了relay log中:而sla

  • IntelliJ IDEA 中使用jRebel进行 Java 热部署教程图解

    jrebel JRebel是一套JavaEE开发工具.JRebel允许开发团队在有限的时间内完成更多的任务修正更多的问题,发布更高质量的软件产品. JRebel是收费软件,用户可以在JRebel官方站点下载30天的评估版本. Jrebel 可快速实现热部署,节省了大量重启时间,提高了个人开发效率. Rebel是一款JAVA虚拟机插件,它使得JAVA程序员能在不进行重部署的情况下,即时看到代码的改变对一个应用程序带来的影响.JRebel使你能即时分别看到代码.类和资源的变化,你可以一个个地上传而不

  • PostgreSQL中常用的时间日期脚本使用教程

    获取系统时间函数 select now(); --2013-11-28 16:20:25.259715+08 select current_timestamp; --2013-11-28 16:20:38.815466+08 select current_date; --2013-11-28 select current_time; --16:21:08.981171+08 时间的计算 --使用interval select now()+interval '2 day'; --2013-11-3

  • ZooKeeper的安装及部署教程

    一.Zookeeper介绍 •是一个针对大型分布式系统的可靠协调系统: •提供的功能包括:配置维护.名字服务.分布式同步.组服务等: •目标就是封装好复杂易出错的关键职务,将简单易用的接口和性能高效.功能稳定的系统提供给用户: •Zookeeper已经成为Hadoop生态系统中的基础组件. 二.Zookeeper特点 •最终一致性:为客户端展示同一视图,这是Zookeeper最重要的性能: •可靠性:如果消息被一台服务器接受,那么它将被所有的服务器接受: •原子性:更新只能成功或失败,没有中间状

  • Docker中搭建FastDFS文件系统(多图教程)

    关于FastDFS FastDFS 是以 C 语言开发的一项开源轻量级分布式文件系统,他对文件进行管理,主要功能有:文件存储,文件同步,文件访问(文件上传/下载)等,特别适合以文件为载体的在线服务,如:图片网站,视频网站等. 外话:FastDFS 是阿里余庆大神做的一个个人项目,从2008年研发FastDFS开源至今,在一些互联网创业公司中备受推崇,GitHub开源地址:https://github.com/happyfish100/fastdfs 本文基于Docker搭建,以下是正文内容: 1

  • Python开发之Nginx+uWSGI+virtualenv多项目部署教程

    1.新建独立运行环境,命名为env [root@vultr ~]# mkdir projects # 测试的项目总目录 [root@vultr ~]# pip3 install virtualenv [root@vultr ~]# cd projects [root@vultr projects]# virtualenv env --python=python3 --no-site-packages --python:指定Python版本 --no-site-packages:不复制系统已安装P

  • 在Docker中安装Elasticsearch7.6.2的教程

    安装docker 你要安装完成docker,不再说明了. 安装Elasticsearch 注意:使用版本为 7.6.2,你可以选择其他版本 拉取镜像 docker pull elasticsearch:7.6.2 启动容器 docker run --restart=always -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" \ -e ES_JAVA_OPTS="-Xms512m -Xmx512m"

  • 浅谈PostgreSQL中的孤儿文件用法(orphaned data files)

    创建一个测试表 postgres=# create table t1(a int); CREATE TABLE postgres=# select pg_relation_filepath('t1'); pg_relation_filepath ---------------------- base/75062/75297 (1 row) postgres=# 在操作系统上已经可以看到该文件. $ ls -la $PGDATA/base/75062/75297 -rw------- 1 post

  • mariadb的主从复制、主主复制、半同步复制配置详解

    主从服务器的时间要同步,数据库版本最好是一致的,以免造成函数处理.日志读取.日志解析等发生异常. 以下三个主从复制的设置是独立的. 注意防火墙和selinux的影响. 1.简单主从复制的实现 (1)主服务器的配置 1)安装mariadb-server [root@localhost ~]# yum -y install mariadb-server 2)编辑/etc/my.cnf文件 [root@localhost ~]# vim /etc/my.cnf 在[mysqld]段的最后添加以下内容

随机推荐