Native GraphQL Database with graph backend

• [x] Low Latency
• [x] High Throughput
• [x] RDF Parsing
• [x] ~Rocks DB~ Badger DB for persistence
• [x] ~Commit Logs~ (Replaced by RAFT logs)
• [x] Query Language - GraphQL-like
  • [x] Query
  • [x] Root Arguments
  • [x] Fields
  • [x] Response in JSON
  • [x] Field Arguments
  • [x] Field Alias
  • [x] Mutations #23 [v0.2]
  • [x] Fragments #8
  • [x] Variables
  • [x] Type System
    • [x] Scalar Types
    • [x] ~Object Types~
  • [x] Mutation validation for scalar types
  • [x] String matching / Name search
  • [x] Sort by attribute
  • [x] Limit number of results #9
  • [x] Filter
    • [x] anyof
    • [x] allof
    • [x] eq (equal)
    • [x] inequality (>=, <=, >, <)
  • [x] Aggregate Functions
    • [x] count
    • [x] sum
    • [x] max
    • [x] min
  • [x] Geospatial Queries
    • [x] Nearby
    • [x] Within
    • [x] Contains
    • [x] Intersects
• [x] Official Clients
  • [x] Javascript
  • [x] Go [v0.3]
  • [x] Java
• [x] Distributed Transactions
• [x] Distributed #14 [v0.2]
  • [x] Distributed Loader [v0.2]
  • [x] Distributed Server [v0.2]
• [x] Clustering
  • [x] Node discovery and membership via Dgraph Zero
• [x] High Availability
  • [x] Raft
  • [x] Automatic Data Replication
  • [x] Automatic Failover for reads
  • [x] Read linearizability
• [x] Resilience
  • [x] Shard moves to handle server failure
• [x] Export
• [ ] Backup

After v1.0 / Proprietary Plugins

• [ ] Multi-homing support
• [ ] Cypher Support
• [ ] Access Control Lists
• [ ] Query Graphical User Interface
• [ ] User authentication
• [ ] Cluster Management
• [ ] SPARQL [maybe]
• [ ] Tinkerpop Support [maybe]
• [x] Distributed transactions [maybe]

Here's the product roadmap for 2019:

• [x] Stronger Type System
• [ ] Support official GraphQL spec natively (#933).
• [ ] Live streaming of updated responses
• [ ] Gremlin support
• [x] Full JSON support across exports, bulk loading, and live loading.
• [x] Upserts

Enterprise features:

• [x] Binary backups
  • [x] Full Backups
  • [x] Incremental Backups
• [x] Access Control Lists
• [ ] Audit Logs
• [x] Encryption at Rest
• [ ] Dgraph cluster running across remote regions
• [ ] Point in time recovery

Tell us what more you'd like to see happen in 2019!

Here's the product roadmap for 2020.

• [ ] Official GraphQL spec compliance
  • [x] Queries (Q1)
  • [x] Mutations (Q1)
  • [ ] Subscriptions (Q2)
  • [ ] Live Queries (Q2)
• [x] SaaS (Q2)
• [ ] Scalability, performance, and reliability
  • [ ] Single predicate sharded across groups (TBD)
  • [ ] Distributed bulk loader (Q3)
  • [ ] Rolling upgrades (Q4)
  • [x] Ludicrous mode (Q1)
  • [ ] Query Planner (TBD)
• [ ] Enterprise features
  • [ ] Change Data Capture (Q3)
  • [ ] Multi-tenancy (Q3)
  • [ ] Point-in-time recovery (TBD)
  • [ ] Audit logs (Q3)
  • [ ] ACL integration with AD, LDAP (stretch goal)
• [ ] Integrations
  • [ ] Kubernetes
    • [ ] Operator (TBD)
    • [ ] Bulk Loader (TBD)
  • [ ] Kafka (TBD)
  • [ ] Others. Let us know in comments.

We have mentioned the features we are planning to focus on in Q1 and Q2 (first half of 2020). For the rest, we'll assess them for their ETA as reach mid-year. Tell us what more you'd like to see happen in 2020!

Work In Progress

• Support Open Cypher
• Support Gremlin
• [ ] Support GraphQL spec
• [ ] Subscriptions
• [ ] Stronger Type System
• [ ] Backup
• [x] Python Driver
• [x] Simplify TLS setup
• [ ] Authentication (full control)
• [x] Jepsen test integration (http://jepsen.io/analyses/dgraph-1-0-2)

I've read #114 and understand why you've chosen not to follow the graphql spec. However, graphql clients like Relay and Apollo provide some very useful features out of the box such as client side caching, optimistic updates, combining queries, etc.

I ended up creating a graphql server to map some basic graphql queries to graphql+-. However, I'm struggling with how to do this efficiently within the graphql server resolve functions without creating a lot of query-specific logic.

Can you please provide an example of the recommended way to interact with graphql clients like Relay and Apollo (either directly or via a graphql server)?

Recently we are doing some research on contact maps and we have a graph consist of almost 2 billion contacts, so obviously we need a graph database. Dgraph is really impressive and really cool. Following the document, we arrange our data into rdf using spark. Now we have about 4.2 billion rdfs.

The schema looks like this: mutation{ schema{ thisphone: string @index(hash) . contact: uid . contact_of: uid . } }

The rdf looks like this: <contact_p104008111111> <contact_of> <contact_p113761083758> (name="sam", ots=1452908610, lts=1501758356, status=1) . <contact_p104008111111> <contact_of> <contact_p113810888226> (name="frank", ots=1453119360, lts=1500729904, status=1) . <contact_p104008111111> <contact_of> <contact_p113811659687> (name="tony", ots=1444992764, lts=1498013559, status=1) .

We are running dgraph on a machine with 64G memory, 3.5T ssd(raid5 though), 40 cores. Now problem is: the import speed will converge to about 20000/s after several minutes. This is not very slow, however when compare to 4.2 billion rdfs still we needs quite a lot of time(3 days). So can we generate sst files and vlogs on spark and then simply copy it to the p directory ? Glad to hear other ways to accelerate the import procedure. Thanks.

If you suspect this could be a bug, follow the template.

• What version of Dgraph are you using?

v0.9.4

• Have you tried reproducing the issue with latest release? yes

• What is the hardware spec (RAM, OS)? macOs Sierra 10.12.6

• Expected behaviour and actual result.

When I run dgraph server with the binaries version v0.9.4, i run the java client such as

Mutation.Builder b = Mutation.newBuilder(mu);
b.addDel(
                  NQuad.newBuilder()
                            .setSubject(uid)
                            .setPredicate(predicate)
                            .setObjectId(valueUid)            // valueUid is the Object's uid
                            .build()
).build();

the server throw an exeption such as

Exception in thread "main" io.grpc.StatusRuntimeException: UNKNOWN: Please use * with delete operation for non-list type
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:210)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:191)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:124)
	at io.dgraph.DgraphGrpc$DgraphBlockingStub.mutate(DgraphGrpc.java:298)
	at io.dgraph.DgraphClient$Transaction.mutate(DgraphClient.java:211)

but when i run dgraph with the source code , i success delete the edge use the same java code

can anyone tell me why. (is the binaries version is not use the newest source code?)

This is the most popular page on discuss: https://discuss.dgraph.io/t/differences-between-dgraph-and-cayley/23

This is also the most popular question on Reddit: https://www.reddit.com/comments/4fs7qm/why_we_built_dgraph/

And again: https://www.reddit.com/r/golang/comments/3uyj8m/announcing_alpha_release_of_open_source/

And again on HN: https://news.ycombinator.com/item?id=11322444

So, it's about time we do a real benchmark. @ashwin95r has written some code to benchmark Dgraph for his final year thesis. So, he will be the coordinator for this project.

We can reuse that code to throw similar queries at Cayley and see how it compares.

If you suspect this could be a bug, follow the template.

• What version of Dgraph are you using? docker dgraph/dgraph:v1.0.5

• Have you tried reproducing the issue with latest release? Yes, I tried it on dgraph/dgraph:master

• What is the hardware spec (RAM, OS)? I run kubernetes cluster on AWS. Instance is m5.xlarge with 16 GB of RAM and 250 SSD HD

• Steps to reproduce the issue (command/config used to run Dgraph).

1. Start kubernetes cluster with configuration:

# There are 4 public services exposed, users can use:
#       dgraph-zero-public - To load data using Live & Bulk Loaders
#       dgraph-server-public - To connect clients and for HTTP APIs
#       dgraph-ratel-public - For Dgraph UI
#       dgraph-server-x-http-public - Use for debugging & profiling
apiVersion: v1
kind: Service
metadata:
  name: dgraph-zero-public
  labels:
    app: dgraph-zero
spec:
  type: LoadBalancer
  ports:
  - port: 5080
    targetPort: 5080
    name: zero-grpc
  - port: 6080
    targetPort: 6080
    name: zero-http
  selector:
    app: dgraph-zero
---
apiVersion: v1
kind: Service
metadata:
  name: dgraph-server-public
  labels:
    app: dgraph-server
spec:
  type: LoadBalancer
  ports:
  - port: 8080
    targetPort: 8080
    name: server-http
  - port: 9080
    targetPort: 9080
    name: server-grpc
  selector:
    app: dgraph-server
---
# This service is created in-order to debug & profile a specific server.
# You can create one for each server that you need to profile.
# For a more general HTTP APIs use the above service instead.
apiVersion: v1
kind: Service
metadata:
  name: dgraph-ratel-public
  labels:
    app: dgraph-ratel
spec:
  type: NodePort
  ports:
  - port: 8000
    targetPort: 8000
    name: ratel-http
  selector:
    app: dgraph-ratel
---
# This is a headless service which is neccessary for discovery for a dgraph-zero StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
  name: dgraph-zero
  labels:
    app: dgraph-zero
spec:
  ports:
  - port: 5080
    targetPort: 5080
    name: zero-grpc
  clusterIP: None
  selector:
    app: dgraph-zero
---
# This is a headless service which is neccessary for discovery for a dgraph-server StatefulSet.
# https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
apiVersion: v1
kind: Service
metadata:
  name: dgraph-server
  labels:
    app: dgraph-server
spec:
  ports:
  - port: 7080
    targetPort: 7080
    name: server-grpc-int
  clusterIP: None
  selector:
    app: dgraph-server
---
# This StatefulSet runs 3 Dgraph Zero.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dgraph-zero
spec:
  serviceName: "dgraph-zero"
  replicas: 1
  selector:
    matchLabels:
      app: dgraph-zero
  template:
    metadata:
      labels:
        app: dgraph-zero
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - dgraph-zero
              topologyKey: kubernetes.io/hostname
      containers:
      - name: zero
        image: dgraph/dgraph:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5080
          name: zero-grpc
        - containerPort: 6080
          name: zero-http
        volumeMounts:
        - name: datadir
          mountPath: /dgraph
        command:
          - bash
          - "-c"
          - |
            set -ex
            [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
            ordinal=${BASH_REMATCH[1]}
            idx=$(($ordinal + 1))
            if [[ $ordinal -eq 0 ]]; then
              dgraph zero --my=$(hostname -f):5080 --idx $idx --replicas 3
            else
              dgraph zero --my=$(hostname -f):5080 --peer dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080 --idx $idx --replicas 3
            fi
      terminationGracePeriodSeconds: 60
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: datadir
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          storage: 5Gi
---
# This StatefulSet runs 6 Dgraph Server forming two server groups, 3 servers in each group.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dgraph-server
spec:
  serviceName: "dgraph-server"
  replicas: 0
  selector:
    matchLabels:
      app: dgraph-server
  template:
    metadata:
      labels:
        app: dgraph-server
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - dgraph-server
              topologyKey: kubernetes.io/hostname
      containers:
      - name: server
        image: dgraph/dgraph:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 7080
          name: server-grpc-int
        - containerPort: 8080
          name: server-http
        - containerPort: 9080
          name: server-grpc
        volumeMounts:
        - name: datadir
          mountPath: /dgraph
        command:
          - bash
          - "-c"
          - |
            set -ex
            dgraph server --my=$(hostname -f):7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
      terminationGracePeriodSeconds: 60
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: datadir
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          storage: 250Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dgraph-ratel
  labels:
    app: dgraph-ratel
spec:
  selector:
    matchLabels:
      app: dgraph-ratel
  template:
    metadata:
      labels:
        app: dgraph-ratel
    spec:
      containers:
      - name: ratel
        image: dgraph/dgraph:latest
        ports:
        - containerPort: 8000
        command:
          - dgraph-ratel

2. When cluster is ready, I start dgraph bulk on separate machine with 32 GB of RAM. Bulk process is connected to k8s zero server and finish with

MAP 01h34m47s rdf_count:465.1M rdf_speed:81.77k/sec edge_count:1.560G edge_speed:274.4k/sec
...
REDUCE 03h11m37s [100.00%] edge_count:1.560G edge_speed:268.7k/sec plist_count:641.3M plist_speed:110.4k/sec

It produce about 111 GB.

3. I create k8s PVC SSD HDD (250 GB) for dgraph server 0 and copy bulk data on it.
4. Setup replication count for dgraph server to 1.
5. Wait for boot on.

kubectl logs -f dgraph-server-0
++ hostname -f
+ dgraph server --my=dgraph-server-0.dgraph-server.default.svc.cluster.local:7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 13:28:24 worker.go:89: Worker listening at address: [::]:7080
2018/06/01 13:28:24 gRPC server started.  Listening on port 9080
2018/06/01 13:28:24 HTTP server started.  Listening on port 8080
2018/06/01 13:28:24 groups.go:78: Current Raft Id: 0
2018/06/01 13:28:25 pool.go:108: == CONNECT ==> Setting dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 13:28:25 groups.go:105: Connected to group zero. Assigned group: 1
2018/06/01 13:28:25 draft.go:170: Node ID: 1 with GroupID: 1
2018/06/01 13:28:25 node.go:240: Group 1 found 0 entries
2018/06/01 13:28:25 draft.go:937: New Node for group: 1
2018/06/01 13:28:25 raft.go:567: INFO: 1 became follower at term 0
2018/06/01 13:28:25 raft.go:315: INFO: newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018/06/01 13:28:25 raft.go:567: INFO: 1 became follower at term 1
2018/06/01 13:28:25 node.go:118: Setting conf state to nodes:1 
2018/06/01 13:28:25 raft.go:749: INFO: 1 is starting a new election at term 1
2018/06/01 13:28:25 groups.go:346: Serving tablet for: _predicate_
2018/06/01 13:28:25 raft.go:580: INFO: 1 became candidate at term 2
2018/06/01 13:28:25 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 2
2018/06/01 13:28:25 raft.go:621: INFO: 1 became leader at term 2
2018/06/01 13:28:25 node.go:301: INFO: raft.node: 1 elected leader 1 at term 2
2018/06/01 13:28:25 groups.go:346: Serving tablet for: at
2018/06/01 13:28:25 mutation.go:181: Done schema update predicate:"_predicate_" value_type:STRING list:true 
2018/06/01 13:28:25 groups.go:346: Serving tablet for: in
2018/06/01 13:28:25 groups.go:346: Serving tablet for: count
...
hundred of predicates
...
2018/06/01 13:35:07 groups.go:346: Serving tablet for: user.selected.organization
2018/06/01 13:35:07 groups.go:346: Serving tablet for: user.preferred.organization
2018/06/01 13:35:07 groups.go:346: Serving tablet for: notification.occurrence.date
2018/06/01 13:35:07 groups.go:346: Serving tablet for: dummy

Server is responding and everything looks good.

6. Replicas count do dgraph server is set on 2. Snapshot is sending:

2018/06/01 15:12:45 raft.go:749: INFO: 1 is starting a new election at term 2
2018/06/01 15:12:45 raft.go:580: INFO: 1 became candidate at term 3
2018/06/01 15:12:45 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 3
2018/06/01 15:12:45 raft.go:621: INFO: 1 became leader at term 3
2018/06/01 15:12:45 node.go:301: INFO: raft.node: 1 elected leader 1 at term 3
2018/06/01 17:45:11 pool.go:158: Echo error from dgraph-server-1.dgraph-server.default.svc.cluster.local:7080. Err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/06/01 17:45:11 pool.go:108: == CONNECT ==> Setting dgraph-server-1.dgraph-server.default.svc.cluster.local:7080
2018/06/01 17:45:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05s, bytes sent: 22 MB, bytes/sec 4393253
2018/06/01 17:45:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10s, bytes sent: 44 MB, bytes/sec 4449208
2018/06/01 17:45:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15s, bytes sent: 66 MB, bytes/sec 4404129
2018/06/01 17:45:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20s, bytes sent: 88 MB, bytes/sec 4381694
2018/06/01 17:45:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25s, bytes sent: 110 MB, bytes/sec 4391688
2018/06/01 17:45:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30s, bytes sent: 131 MB, bytes/sec 4377680
2018/06/01 17:45:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 35s, bytes sent: 152 MB, bytes/sec 4347668
2018/06/01 17:45:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 40s, bytes sent: 174 MB, bytes/sec 4358675
2018/06/01 17:45:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 45s, bytes sent: 194 MB, bytes/sec 4318528
2018/06/01 17:46:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 50s, bytes sent: 216 MB, bytes/sec 4320323
2018/06/01 17:46:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 55s, bytes sent: 237 MB, bytes/sec 4301221
2018/06/01 17:46:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m00s, bytes sent: 257 MB, bytes/sec 4282766
2018/06/01 17:46:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m05s, bytes sent: 279 MB, bytes/sec 4288907
2018/06/01 17:46:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m10s, bytes sent: 299 MB, bytes/sec 4266223
2018/06/01 17:46:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m15s, bytes sent: 321 MB, bytes/sec 4276578
2018/06/01 17:46:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m20s, bytes sent: 340 MB, bytes/sec 4255449
2018/06/01 17:46:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m25s, bytes sent: 362 MB, bytes/sec 4253578
2018/06/01 17:46:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m30s, bytes sent: 383 MB, bytes/sec 4251331
2018/06/01 17:46:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m35s, bytes sent: 402 MB, bytes/sec 4229044
2018/06/01 17:46:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m40s, bytes sent: 423 MB, bytes/sec 4234240
2018/06/01 17:46:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m45s, bytes sent: 445 MB, bytes/sec 4235367
2018/06/01 17:47:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m50s, bytes sent: 463 MB, bytes/sec 4208357
2018/06/01 17:47:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m55s, bytes sent: 484 MB, bytes/sec 4205258
2018/06/01 17:47:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m00s, bytes sent: 505 MB, bytes/sec 4208297
2018/06/01 17:47:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m05s, bytes sent: 524 MB, bytes/sec 4193215
2018/06/01 17:47:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m10s, bytes sent: 543 MB, bytes/sec 4179465
2018/06/01 17:47:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m15s, bytes sent: 564 MB, bytes/sec 4177842
2018/06/01 17:47:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m20s, bytes sent: 584 MB, bytes/sec 4169674
2018/06/01 17:47:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m25s, bytes sent: 604 MB, bytes/sec 4162970
2018/06/01 17:47:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m30s, bytes sent: 622 MB, bytes/sec 4143588
2018/06/01 17:47:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m35s, bytes sent: 642 MB, bytes/sec 4140368
2018/06/01 17:47:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m40s, bytes sent: 661 MB, bytes/sec 4133150
2018/06/01 17:47:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m45s, bytes sent: 681 MB, bytes/sec 4124900
2018/06/01 17:48:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m50s, bytes sent: 701 MB, bytes/sec 4121609
2018/06/01 17:48:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m55s, bytes sent: 717 MB, bytes/sec 4097400
2018/06/01 17:48:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m00s, bytes sent: 736 MB, bytes/sec 4088239
2018/06/01 17:48:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m05s, bytes sent: 756 MB, bytes/sec 4084180
2018/06/01 17:48:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m10s, bytes sent: 774 MB, bytes/sec 4072266
2018/06/01 17:48:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m15s, bytes sent: 791 MB, bytes/sec 4055813
2018/06/01 17:48:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m19s, bytes sent: 811 MB, bytes/sec 4073739
2018/06/01 17:48:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m25s, bytes sent: 830 MB, bytes/sec 4050333
2018/06/01 17:48:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m30s, bytes sent: 850 MB, bytes/sec 4046088
2018/06/01 17:48:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m35s, bytes sent: 869 MB, bytes/sec 4043575
2018/06/01 17:48:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m40s, bytes sent: 886 MB, bytes/sec 4025607
2018/06/01 17:48:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m45s, bytes sent: 902 MB, bytes/sec 4011002
2018/06/01 17:49:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m50s, bytes sent: 921 MB, bytes/sec 4004382
2018/06/01 17:49:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m55s, bytes sent: 939 MB, bytes/sec 3997608
2018/06/01 17:49:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m00s, bytes sent: 958 MB, bytes/sec 3991401
2018/06/01 17:49:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m05s, bytes sent: 977 MB, bytes/sec 3986740
2018/06/01 17:49:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m10s, bytes sent: 996 MB, bytes/sec 3982052
2018/06/01 17:49:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m15s, bytes sent: 1.0 GB, bytes/sec 3977189
2018/06/01 17:49:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m20s, bytes sent: 1.0 GB, bytes/sec 3973324
2018/06/01 17:49:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m25s, bytes sent: 1.1 GB, bytes/sec 3970729
2018/06/01 17:49:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m30s, bytes sent: 1.1 GB, bytes/sec 3968236
2018/06/01 17:49:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m35s, bytes sent: 1.1 GB, bytes/sec 3965634
2018/06/01 17:49:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m40s, bytes sent: 1.1 GB, bytes/sec 3962986
2018/06/01 17:49:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m45s, bytes sent: 1.1 GB, bytes/sec 3950007
2018/06/01 17:50:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m50s, bytes sent: 1.1 GB, bytes/sec 3949570
2018/06/01 17:50:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m55s, bytes sent: 1.2 GB, bytes/sec 3948760
2018/06/01 17:50:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m00s, bytes sent: 1.2 GB, bytes/sec 3943994
2018/06/01 17:50:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m05s, bytes sent: 1.2 GB, bytes/sec 3938128
2018/06/01 17:50:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m10s, bytes sent: 1.2 GB, bytes/sec 3931558
2018/06/01 17:50:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m15s, bytes sent: 1.2 GB, bytes/sec 3927464
2018/06/01 17:50:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m20s, bytes sent: 1.3 GB, bytes/sec 3924201
2018/06/01 17:50:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m25s, bytes sent: 1.3 GB, bytes/sec 3919021
2018/06/01 17:50:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m30s, bytes sent: 1.3 GB, bytes/sec 3913088
2018/06/01 17:50:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m35s, bytes sent: 1.3 GB, bytes/sec 3908579
2018/06/01 17:50:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m40s, bytes sent: 1.3 GB, bytes/sec 3901949
2018/06/01 17:50:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m45s, bytes sent: 1.3 GB, bytes/sec 3895386
2018/06/01 17:51:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m50s, bytes sent: 1.4 GB, bytes/sec 3890055
2018/06/01 17:51:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m55s, bytes sent: 1.4 GB, bytes/sec 3890257
2018/06/01 17:51:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m00s, bytes sent: 1.4 GB, bytes/sec 3881119
2018/06/01 17:51:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m05s, bytes sent: 1.4 GB, bytes/sec 3883889
2018/06/01 17:51:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m10s, bytes sent: 1.4 GB, bytes/sec 3878649
2018/06/01 17:51:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m15s, bytes sent: 1.4 GB, bytes/sec 3862716
2018/06/01 17:51:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m20s, bytes sent: 1.5 GB, bytes/sec 3857627
2018/06/01 17:51:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m25s, bytes sent: 1.5 GB, bytes/sec 3850813
2018/06/01 17:51:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m30s, bytes sent: 1.5 GB, bytes/sec 3848179
2018/06/01 17:51:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m35s, bytes sent: 1.5 GB, bytes/sec 3852186
2018/06/01 17:51:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m40s, bytes sent: 1.5 GB, bytes/sec 3847207
2018/06/01 17:51:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m45s, bytes sent: 1.6 GB, bytes/sec 3839918
2018/06/01 17:52:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m50s, bytes sent: 1.6 GB, bytes/sec 3830104
2018/06/01 17:52:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 06m55s, bytes sent: 1.6 GB, bytes/sec 3824781
2018/06/01 17:52:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m00s, bytes sent: 1.6 GB, bytes/sec 3827268
2018/06/01 17:52:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m05s, bytes sent: 1.6 GB, bytes/sec 3823951
2018/06/01 17:52:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m10s, bytes sent: 1.6 GB, bytes/sec 3814258
2018/06/01 17:52:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m15s, bytes sent: 1.7 GB, bytes/sec 3807813
2018/06/01 17:52:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m20s, bytes sent: 1.7 GB, bytes/sec 3796504
2018/06/01 17:52:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m25s, bytes sent: 1.7 GB, bytes/sec 3801479
2018/06/01 17:52:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m30s, bytes sent: 1.7 GB, bytes/sec 3797149
2018/06/01 17:52:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m35s, bytes sent: 1.7 GB, bytes/sec 3784648
2018/06/01 17:52:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m40s, bytes sent: 1.7 GB, bytes/sec 3776229
2018/06/01 17:52:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m45s, bytes sent: 1.8 GB, bytes/sec 3770879
2018/06/01 17:53:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m50s, bytes sent: 1.8 GB, bytes/sec 3775632
2018/06/01 17:53:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 07m55s, bytes sent: 1.8 GB, bytes/sec 3767350
2018/06/01 17:53:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m00s, bytes sent: 1.8 GB, bytes/sec 3758027
2018/06/01 17:53:19 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m07s, bytes sent: 1.8 GB, bytes/sec 3744498
2018/06/01 17:53:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m10s, bytes sent: 1.8 GB, bytes/sec 3721572
2018/06/01 17:53:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m15s, bytes sent: 1.9 GB, bytes/sec 3780391
2018/06/01 17:53:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m19s, bytes sent: 1.9 GB, bytes/sec 3861983
2018/06/01 17:53:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m25s, bytes sent: 1.9 GB, bytes/sec 3860727
2018/06/01 17:53:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m30s, bytes sent: 2.0 GB, bytes/sec 3855591
2018/06/01 17:53:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m35s, bytes sent: 2.0 GB, bytes/sec 3848487
2018/06/01 17:53:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m40s, bytes sent: 2.0 GB, bytes/sec 3841572
2018/06/01 17:53:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m45s, bytes sent: 2.0 GB, bytes/sec 3847627
2018/06/01 17:54:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m50s, bytes sent: 2.0 GB, bytes/sec 3841937
2018/06/01 17:54:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 08m55s, bytes sent: 2.1 GB, bytes/sec 3835490
2018/06/01 17:54:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m00s, bytes sent: 2.1 GB, bytes/sec 3838153
2018/06/01 17:54:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m05s, bytes sent: 2.1 GB, bytes/sec 3836492
2018/06/01 17:54:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m10s, bytes sent: 2.1 GB, bytes/sec 3829414
2018/06/01 17:54:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m15s, bytes sent: 2.1 GB, bytes/sec 3822346
2018/06/01 17:54:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m20s, bytes sent: 2.1 GB, bytes/sec 3828629
2018/06/01 17:54:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m25s, bytes sent: 2.2 GB, bytes/sec 3820716
2018/06/01 17:54:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m30s, bytes sent: 2.2 GB, bytes/sec 3812062
2018/06/01 17:54:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m35s, bytes sent: 2.2 GB, bytes/sec 3817085
2018/06/01 17:54:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m40s, bytes sent: 2.2 GB, bytes/sec 3812566
2018/06/01 17:54:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m45s, bytes sent: 2.2 GB, bytes/sec 3804485
2018/06/01 17:55:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m50s, bytes sent: 2.2 GB, bytes/sec 3803759
2018/06/01 17:55:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m55s, bytes sent: 2.3 GB, bytes/sec 3804736
2018/06/01 17:55:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 09m59s, bytes sent: 2.3 GB, bytes/sec 3797506
2018/06/01 17:55:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m05s, bytes sent: 2.3 GB, bytes/sec 3783498
2018/06/01 17:55:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m10s, bytes sent: 2.3 GB, bytes/sec 3790308
2018/06/01 17:55:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m15s, bytes sent: 2.3 GB, bytes/sec 3781984
2018/06/01 17:55:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m20s, bytes sent: 2.3 GB, bytes/sec 3774783
2018/06/01 17:55:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m25s, bytes sent: 2.4 GB, bytes/sec 3777479
2018/06/01 17:55:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m30s, bytes sent: 2.4 GB, bytes/sec 3769624
2018/06/01 17:55:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m35s, bytes sent: 2.4 GB, bytes/sec 3767215
2018/06/01 17:55:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m40s, bytes sent: 2.4 GB, bytes/sec 3770315
2018/06/01 17:55:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m45s, bytes sent: 2.4 GB, bytes/sec 3762009
2018/06/01 17:56:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m50s, bytes sent: 2.4 GB, bytes/sec 3762667
2018/06/01 17:56:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10m55s, bytes sent: 2.5 GB, bytes/sec 3761090
2018/06/01 17:56:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m00s, bytes sent: 2.5 GB, bytes/sec 3751960
2018/06/01 17:56:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m05s, bytes sent: 2.5 GB, bytes/sec 3757080
2018/06/01 17:56:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m10s, bytes sent: 2.5 GB, bytes/sec 3751188
2018/06/01 17:56:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m15s, bytes sent: 2.5 GB, bytes/sec 3743468
2018/06/01 17:56:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m20s, bytes sent: 2.6 GB, bytes/sec 3750871
2018/06/01 17:56:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m25s, bytes sent: 2.6 GB, bytes/sec 3742799
2018/06/01 17:56:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m30s, bytes sent: 2.6 GB, bytes/sec 3738491
2018/06/01 17:56:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m35s, bytes sent: 2.6 GB, bytes/sec 3738602
2018/06/01 17:56:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m40s, bytes sent: 2.6 GB, bytes/sec 3730983
2018/06/01 17:56:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m45s, bytes sent: 2.6 GB, bytes/sec 3735215
2018/06/01 17:57:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m50s, bytes sent: 2.6 GB, bytes/sec 3730017
2018/06/01 17:57:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 11m55s, bytes sent: 2.7 GB, bytes/sec 3721870
2018/06/01 17:57:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m00s, bytes sent: 2.7 GB, bytes/sec 3728146
2018/06/01 17:57:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m05s, bytes sent: 2.7 GB, bytes/sec 3720046
2018/06/01 17:57:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m10s, bytes sent: 2.7 GB, bytes/sec 3715153
2018/06/01 17:57:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m15s, bytes sent: 2.7 GB, bytes/sec 3715948
2018/06/01 17:57:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m20s, bytes sent: 2.7 GB, bytes/sec 3707757
2018/06/01 17:57:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m25s, bytes sent: 2.8 GB, bytes/sec 3711457
2018/06/01 17:57:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m30s, bytes sent: 2.8 GB, bytes/sec 3706027
2018/06/01 17:57:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m35s, bytes sent: 2.8 GB, bytes/sec 3699451
2018/06/01 17:57:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m40s, bytes sent: 2.8 GB, bytes/sec 3703611
2018/06/01 17:57:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m45s, bytes sent: 2.8 GB, bytes/sec 3694715
2018/06/01 17:58:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m50s, bytes sent: 2.8 GB, bytes/sec 3689339
2018/06/01 17:58:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 12m55s, bytes sent: 2.9 GB, bytes/sec 3687580
2018/06/01 17:58:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m00s, bytes sent: 2.9 GB, bytes/sec 3679697
2018/06/01 17:58:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m05s, bytes sent: 2.9 GB, bytes/sec 3685320
2018/06/01 17:58:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m10s, bytes sent: 2.9 GB, bytes/sec 3677117
2018/06/01 17:58:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m15s, bytes sent: 2.9 GB, bytes/sec 3684752
2018/06/01 17:58:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m20s, bytes sent: 2.9 GB, bytes/sec 3676421
2018/06/01 17:58:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m25s, bytes sent: 3.0 GB, bytes/sec 3676760
2018/06/01 17:58:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m30s, bytes sent: 3.0 GB, bytes/sec 3674590
2018/06/01 17:58:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m35s, bytes sent: 3.0 GB, bytes/sec 3668357
2018/06/01 17:58:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m40s, bytes sent: 3.0 GB, bytes/sec 3673005
2018/06/01 17:58:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m45s, bytes sent: 3.0 GB, bytes/sec 3664252
2018/06/01 17:59:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m50s, bytes sent: 3.0 GB, bytes/sec 3670518
2018/06/01 17:59:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 13m55s, bytes sent: 3.1 GB, bytes/sec 3662421
2018/06/01 17:59:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m00s, bytes sent: 3.1 GB, bytes/sec 3666432
2018/06/01 17:59:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m05s, bytes sent: 3.1 GB, bytes/sec 3660437
2018/06/01 17:59:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m10s, bytes sent: 3.1 GB, bytes/sec 3659825
2018/06/01 17:59:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m15s, bytes sent: 3.1 GB, bytes/sec 3657310
2018/06/01 17:59:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m20s, bytes sent: 3.1 GB, bytes/sec 3652183
2018/06/01 17:59:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m25s, bytes sent: 3.2 GB, bytes/sec 3653679
2018/06/01 17:59:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m30s, bytes sent: 3.2 GB, bytes/sec 3645780
2018/06/01 17:59:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m35s, bytes sent: 3.2 GB, bytes/sec 3651923
2018/06/01 17:59:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m40s, bytes sent: 3.2 GB, bytes/sec 3643520
2018/06/01 17:59:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m45s, bytes sent: 3.2 GB, bytes/sec 3650083
2018/06/01 18:00:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m50s, bytes sent: 3.2 GB, bytes/sec 3641529
2018/06/01 18:00:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14m55s, bytes sent: 3.3 GB, bytes/sec 3643243
2018/06/01 18:00:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m00s, bytes sent: 3.3 GB, bytes/sec 3638431
2018/06/01 18:00:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m05s, bytes sent: 3.3 GB, bytes/sec 3636386
2018/06/01 18:00:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m10s, bytes sent: 3.3 GB, bytes/sec 3635220
2018/06/01 18:00:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m15s, bytes sent: 3.3 GB, bytes/sec 3629649
2018/06/01 18:00:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m20s, bytes sent: 3.3 GB, bytes/sec 3632656
2018/06/01 18:00:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m25s, bytes sent: 3.4 GB, bytes/sec 3624341
2018/06/01 18:00:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m30s, bytes sent: 3.4 GB, bytes/sec 3628998
2018/06/01 18:00:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m35s, bytes sent: 3.4 GB, bytes/sec 3619894
2018/06/01 18:00:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m40s, bytes sent: 3.4 GB, bytes/sec 3624584
2018/06/01 18:00:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m45s, bytes sent: 3.4 GB, bytes/sec 3616038
2018/06/01 18:01:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m50s, bytes sent: 3.4 GB, bytes/sec 3621920
2018/06/01 18:01:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 15m55s, bytes sent: 3.5 GB, bytes/sec 3612809
2018/06/01 18:01:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m00s, bytes sent: 3.5 GB, bytes/sec 3615435
2018/06/01 18:01:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m05s, bytes sent: 3.5 GB, bytes/sec 3609578
2018/06/01 18:01:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m10s, bytes sent: 3.5 GB, bytes/sec 3612228
2018/06/01 18:01:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m15s, bytes sent: 3.5 GB, bytes/sec 3606000
2018/06/01 18:01:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m20s, bytes sent: 3.5 GB, bytes/sec 3598348
2018/06/01 18:01:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m25s, bytes sent: 3.5 GB, bytes/sec 3599619
2018/06/01 18:01:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m30s, bytes sent: 3.6 GB, bytes/sec 3595854
2018/06/01 18:01:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m34s, bytes sent: 3.6 GB, bytes/sec 3599083
2018/06/01 18:01:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m40s, bytes sent: 3.6 GB, bytes/sec 3591672
2018/06/01 18:01:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m45s, bytes sent: 3.6 GB, bytes/sec 3592740
2018/06/01 18:02:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m50s, bytes sent: 3.6 GB, bytes/sec 3587499
2018/06/01 18:02:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 16m55s, bytes sent: 3.6 GB, bytes/sec 3589162
2018/06/01 18:02:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m00s, bytes sent: 3.7 GB, bytes/sec 3584319
2018/06/01 18:02:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m05s, bytes sent: 3.7 GB, bytes/sec 3579268
2018/06/01 18:02:24 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m13s, bytes sent: 3.7 GB, bytes/sec 3596038
2018/06/01 18:02:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m15s, bytes sent: 3.8 GB, bytes/sec 3637856
2018/06/01 18:02:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m20s, bytes sent: 3.8 GB, bytes/sec 3631768
2018/06/01 18:02:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m25s, bytes sent: 3.8 GB, bytes/sec 3642069
2018/06/01 18:02:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m30s, bytes sent: 3.8 GB, bytes/sec 3638064
2018/06/01 18:02:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m35s, bytes sent: 3.8 GB, bytes/sec 3640471
2018/06/01 18:02:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m40s, bytes sent: 3.9 GB, bytes/sec 3645572
2018/06/01 18:02:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m45s, bytes sent: 3.9 GB, bytes/sec 3641293
2018/06/01 18:03:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m50s, bytes sent: 3.9 GB, bytes/sec 3650106
2018/06/01 18:03:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 17m55s, bytes sent: 3.9 GB, bytes/sec 3642771
2018/06/01 18:03:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m00s, bytes sent: 3.9 GB, bytes/sec 3650795
2018/06/01 18:03:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m05s, bytes sent: 4.0 GB, bytes/sec 3651665
2018/06/01 18:03:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m10s, bytes sent: 4.0 GB, bytes/sec 3652878
2018/06/01 18:03:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m15s, bytes sent: 4.0 GB, bytes/sec 3656737
2018/06/01 18:03:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m20s, bytes sent: 4.0 GB, bytes/sec 3650280
2018/06/01 18:03:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m25s, bytes sent: 4.0 GB, bytes/sec 3659106
2018/06/01 18:03:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m30s, bytes sent: 4.1 GB, bytes/sec 3652469
2018/06/01 18:03:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m35s, bytes sent: 4.1 GB, bytes/sec 3660269
2018/06/01 18:03:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m40s, bytes sent: 4.1 GB, bytes/sec 3662390
2018/06/01 18:03:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m45s, bytes sent: 4.1 GB, bytes/sec 3662130
2018/06/01 18:04:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m50s, bytes sent: 4.1 GB, bytes/sec 3670620
2018/06/01 18:04:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 18m55s, bytes sent: 4.2 GB, bytes/sec 3662772
2018/06/01 18:04:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m00s, bytes sent: 4.2 GB, bytes/sec 3672566
2018/06/01 18:04:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m05s, bytes sent: 4.2 GB, bytes/sec 3672128
2018/06/01 18:04:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m10s, bytes sent: 4.2 GB, bytes/sec 3673396
2018/06/01 18:04:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m15s, bytes sent: 4.3 GB, bytes/sec 3682707
2018/06/01 18:04:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m20s, bytes sent: 4.3 GB, bytes/sec 3674007
2018/06/01 18:04:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m25s, bytes sent: 4.3 GB, bytes/sec 3678505
2018/06/01 18:04:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m30s, bytes sent: 4.3 GB, bytes/sec 3682463
2018/06/01 18:04:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m35s, bytes sent: 4.3 GB, bytes/sec 3679502
2018/06/01 18:04:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m40s, bytes sent: 4.4 GB, bytes/sec 3687799
2018/06/01 18:04:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m45s, bytes sent: 4.4 GB, bytes/sec 3682375
2018/06/01 18:05:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m50s, bytes sent: 4.4 GB, bytes/sec 3688005
2018/06/01 18:05:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 19m55s, bytes sent: 4.4 GB, bytes/sec 3695026
2018/06/01 18:05:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m00s, bytes sent: 4.4 GB, bytes/sec 3686564
2018/06/01 18:05:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m05s, bytes sent: 4.4 GB, bytes/sec 3691981
2018/06/01 18:05:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m10s, bytes sent: 4.5 GB, bytes/sec 3690294
2018/06/01 18:05:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m15s, bytes sent: 4.5 GB, bytes/sec 3689841
2018/06/01 18:05:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m20s, bytes sent: 4.5 GB, bytes/sec 3694863
2018/06/01 18:05:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m25s, bytes sent: 4.5 GB, bytes/sec 3687340
2018/06/01 18:05:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m30s, bytes sent: 4.5 GB, bytes/sec 3695152
2018/06/01 18:05:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m35s, bytes sent: 4.6 GB, bytes/sec 3699043
2018/06/01 18:05:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m40s, bytes sent: 4.6 GB, bytes/sec 3696077
2018/06/01 18:05:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m45s, bytes sent: 4.6 GB, bytes/sec 3704682
2018/06/01 18:06:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m50s, bytes sent: 4.6 GB, bytes/sec 3697894
2018/06/01 18:06:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20m55s, bytes sent: 4.6 GB, bytes/sec 3700931
2018/06/01 18:06:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m00s, bytes sent: 4.7 GB, bytes/sec 3703253
2018/06/01 18:06:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m05s, bytes sent: 4.7 GB, bytes/sec 3695026
2018/06/01 18:06:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m10s, bytes sent: 4.7 GB, bytes/sec 3701953
2018/06/01 18:06:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m15s, bytes sent: 4.7 GB, bytes/sec 3703324
2018/06/01 18:06:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m20s, bytes sent: 4.7 GB, bytes/sec 3702260
2018/06/01 18:06:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m25s, bytes sent: 4.8 GB, bytes/sec 3710631
2018/06/01 18:06:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m30s, bytes sent: 4.8 GB, bytes/sec 3703069
2018/06/01 18:06:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m35s, bytes sent: 4.8 GB, bytes/sec 3709731
2018/06/01 18:06:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m40s, bytes sent: 4.8 GB, bytes/sec 3713884
2018/06/01 18:06:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m45s, bytes sent: 4.8 GB, bytes/sec 3710874
2018/06/01 18:07:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m50s, bytes sent: 4.9 GB, bytes/sec 3714888
2018/06/01 18:07:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 21m55s, bytes sent: 4.9 GB, bytes/sec 3711779
2018/06/01 18:07:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m00s, bytes sent: 4.9 GB, bytes/sec 3710852
2018/06/01 18:07:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m05s, bytes sent: 4.9 GB, bytes/sec 3718597
2018/06/01 18:07:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m10s, bytes sent: 4.9 GB, bytes/sec 3710743
2018/06/01 18:07:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m15s, bytes sent: 5.0 GB, bytes/sec 3718391
2018/06/01 18:07:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m20s, bytes sent: 5.0 GB, bytes/sec 3720274
2018/06/01 18:07:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m25s, bytes sent: 5.0 GB, bytes/sec 3713716
2018/06/01 18:07:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m30s, bytes sent: 5.0 GB, bytes/sec 3716661
2018/06/01 18:07:57 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m35s, bytes sent: 5.0 GB, bytes/sec 3713399
2018/06/01 18:07:57 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m45s, bytes sent: 5.0 GB, bytes/sec 3686194
2018/06/01 18:07:57 node.go:400: WARN: A tick missed to fire. Node blocks too long!
2018/06/01 18:08:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m50s, bytes sent: 5.0 GB, bytes/sec 3680665
2018/06/01 18:08:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 22m55s, bytes sent: 5.1 GB, bytes/sec 3686765
2018/06/01 18:08:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m00s, bytes sent: 5.1 GB, bytes/sec 3688270
2018/06/01 18:08:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m05s, bytes sent: 5.1 GB, bytes/sec 3681854
2018/06/01 18:08:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m10s, bytes sent: 5.1 GB, bytes/sec 3687864
2018/06/01 18:08:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m15s, bytes sent: 5.1 GB, bytes/sec 3688813
2018/06/01 18:08:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m20s, bytes sent: 5.2 GB, bytes/sec 3685633
2018/06/01 18:08:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m25s, bytes sent: 5.2 GB, bytes/sec 3692948
2018/06/01 18:08:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m30s, bytes sent: 5.2 GB, bytes/sec 3689993
2018/06/01 18:08:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m35s, bytes sent: 5.2 GB, bytes/sec 3691985
2018/06/01 18:08:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m40s, bytes sent: 5.3 GB, bytes/sec 3697662
2018/06/01 18:08:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m45s, bytes sent: 5.3 GB, bytes/sec 3691618
2018/06/01 18:09:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m50s, bytes sent: 5.3 GB, bytes/sec 3696240
2018/06/01 18:09:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 23m55s, bytes sent: 5.3 GB, bytes/sec 3703435
2018/06/01 18:09:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m00s, bytes sent: 5.3 GB, bytes/sec 3694984
2018/06/01 18:09:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m05s, bytes sent: 5.3 GB, bytes/sec 3702277
2018/06/01 18:09:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m10s, bytes sent: 5.4 GB, bytes/sec 3707766
2018/06/01 18:09:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m15s, bytes sent: 5.4 GB, bytes/sec 3701348
2018/06/01 18:09:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m20s, bytes sent: 5.4 GB, bytes/sec 3708834
2018/06/01 18:09:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m25s, bytes sent: 5.4 GB, bytes/sec 3710369
2018/06/01 18:09:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m30s, bytes sent: 5.5 GB, bytes/sec 3708364
2018/06/01 18:09:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m35s, bytes sent: 5.5 GB, bytes/sec 3711508
2018/06/01 18:09:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m40s, bytes sent: 5.5 GB, bytes/sec 3709659
2018/06/01 18:09:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m45s, bytes sent: 5.5 GB, bytes/sec 3709562
2018/06/01 18:10:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m50s, bytes sent: 5.5 GB, bytes/sec 3715472
2018/06/01 18:10:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 24m55s, bytes sent: 5.5 GB, bytes/sec 3709425
2018/06/01 18:10:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m00s, bytes sent: 5.6 GB, bytes/sec 3713845
2018/06/01 18:10:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m05s, bytes sent: 5.6 GB, bytes/sec 3717552
2018/06/01 18:10:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m10s, bytes sent: 5.6 GB, bytes/sec 3709310
2018/06/01 18:10:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m15s, bytes sent: 5.6 GB, bytes/sec 3715326
2018/06/01 18:10:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m20s, bytes sent: 5.7 GB, bytes/sec 3721215
2018/06/01 18:10:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m25s, bytes sent: 5.7 GB, bytes/sec 3713876
2018/06/01 18:10:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m30s, bytes sent: 5.7 GB, bytes/sec 3720092
2018/06/01 18:10:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m35s, bytes sent: 5.7 GB, bytes/sec 3721989
2018/06/01 18:10:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m40s, bytes sent: 5.7 GB, bytes/sec 3718912
2018/06/01 18:10:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m45s, bytes sent: 5.8 GB, bytes/sec 3724028
2018/06/01 18:11:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m50s, bytes sent: 5.8 GB, bytes/sec 3721840
2018/06/01 18:11:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25m55s, bytes sent: 5.8 GB, bytes/sec 3720416
2018/06/01 18:11:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m00s, bytes sent: 5.8 GB, bytes/sec 3727138
2018/06/01 18:11:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m05s, bytes sent: 5.8 GB, bytes/sec 3724077
2018/06/01 18:11:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m10s, bytes sent: 5.8 GB, bytes/sec 3724611
2018/06/01 18:11:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m15s, bytes sent: 5.9 GB, bytes/sec 3731143
2018/06/01 18:11:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m20s, bytes sent: 5.9 GB, bytes/sec 3725705
2018/06/01 18:11:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m25s, bytes sent: 5.9 GB, bytes/sec 3729426
2018/06/01 18:11:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m30s, bytes sent: 5.9 GB, bytes/sec 3730807
2018/06/01 18:11:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m35s, bytes sent: 5.9 GB, bytes/sec 3724076
2018/06/01 18:11:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m40s, bytes sent: 6.0 GB, bytes/sec 3726924
2018/06/01 18:11:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m45s, bytes sent: 6.0 GB, bytes/sec 3729892
2018/06/01 18:12:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m50s, bytes sent: 6.0 GB, bytes/sec 3723104
2018/06/01 18:12:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 26m55s, bytes sent: 6.0 GB, bytes/sec 3727931
2018/06/01 18:12:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m00s, bytes sent: 6.0 GB, bytes/sec 3734131
2018/06/01 18:12:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m05s, bytes sent: 6.1 GB, bytes/sec 3727231
2018/06/01 18:12:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m10s, bytes sent: 6.1 GB, bytes/sec 3731079
2018/06/01 18:12:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m15s, bytes sent: 6.1 GB, bytes/sec 3734923
2018/06/01 18:12:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m20s, bytes sent: 6.1 GB, bytes/sec 3729985
2018/06/01 18:12:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m25s, bytes sent: 6.1 GB, bytes/sec 3731835
2018/06/01 18:12:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m30s, bytes sent: 6.2 GB, bytes/sec 3738031
2018/06/01 18:12:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m35s, bytes sent: 6.2 GB, bytes/sec 3731600
2018/06/01 18:12:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m40s, bytes sent: 6.2 GB, bytes/sec 3731866
2018/06/01 18:12:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m45s, bytes sent: 6.2 GB, bytes/sec 3738084
2018/06/01 18:13:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m50s, bytes sent: 6.2 GB, bytes/sec 3734261
2018/06/01 18:13:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 27m55s, bytes sent: 6.3 GB, bytes/sec 3735015
2018/06/01 18:13:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m00s, bytes sent: 6.3 GB, bytes/sec 3734983
2018/06/01 18:13:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m05s, bytes sent: 6.3 GB, bytes/sec 3734066
2018/06/01 18:13:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m10s, bytes sent: 6.3 GB, bytes/sec 3731217
2018/06/01 18:13:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m15s, bytes sent: 6.3 GB, bytes/sec 3736654
2018/06/01 18:13:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m20s, bytes sent: 6.4 GB, bytes/sec 3736024
2018/06/01 18:13:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m25s, bytes sent: 6.4 GB, bytes/sec 3733827
2018/06/01 18:13:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m30s, bytes sent: 6.4 GB, bytes/sec 3735442
2018/06/01 18:13:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m35s, bytes sent: 6.4 GB, bytes/sec 3736446
2018/06/01 18:13:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m40s, bytes sent: 6.4 GB, bytes/sec 3730353
2018/06/01 18:13:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m45s, bytes sent: 6.4 GB, bytes/sec 3736661
2018/06/01 18:14:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m50s, bytes sent: 6.5 GB, bytes/sec 3739884
2018/06/01 18:14:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 28m55s, bytes sent: 6.5 GB, bytes/sec 3733970
2018/06/01 18:14:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m00s, bytes sent: 6.5 GB, bytes/sec 3737049
2018/06/01 18:14:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m05s, bytes sent: 6.5 GB, bytes/sec 3742519
2018/06/01 18:14:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m10s, bytes sent: 6.5 GB, bytes/sec 3734867
2018/06/01 18:14:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m15s, bytes sent: 6.6 GB, bytes/sec 3741173
2018/06/01 18:14:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m20s, bytes sent: 6.6 GB, bytes/sec 3745944
2018/06/01 18:14:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m25s, bytes sent: 6.6 GB, bytes/sec 3738840
2018/06/01 18:14:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m30s, bytes sent: 6.6 GB, bytes/sec 3744760
2018/06/01 18:14:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m35s, bytes sent: 6.7 GB, bytes/sec 3749055
2018/06/01 18:14:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m40s, bytes sent: 6.7 GB, bytes/sec 3743877
2018/06/01 18:14:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m45s, bytes sent: 6.7 GB, bytes/sec 3750013
2018/06/01 18:15:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m50s, bytes sent: 6.7 GB, bytes/sec 3754006
2018/06/01 18:15:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 29m55s, bytes sent: 6.7 GB, bytes/sec 3746860
2018/06/01 18:15:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m00s, bytes sent: 6.8 GB, bytes/sec 3751817
2018/06/01 18:15:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m05s, bytes sent: 6.8 GB, bytes/sec 3757730
2018/06/01 18:15:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m10s, bytes sent: 6.8 GB, bytes/sec 3749419
2018/06/01 18:15:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m15s, bytes sent: 6.8 GB, bytes/sec 3755456
2018/06/01 18:15:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m20s, bytes sent: 6.8 GB, bytes/sec 3761676
2018/06/01 18:15:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m25s, bytes sent: 6.8 GB, bytes/sec 3753418
2018/06/01 18:15:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m30s, bytes sent: 6.9 GB, bytes/sec 3759134
2018/06/01 18:15:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m35s, bytes sent: 6.9 GB, bytes/sec 3765156
2018/06/01 18:15:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m40s, bytes sent: 6.9 GB, bytes/sec 3758325
2018/06/01 18:15:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m45s, bytes sent: 6.9 GB, bytes/sec 3764308
2018/06/01 18:16:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m49s, bytes sent: 7.0 GB, bytes/sec 3772195
2018/06/01 18:16:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30m55s, bytes sent: 7.0 GB, bytes/sec 3762407
2018/06/01 18:16:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 31m00s, bytes sent: 7.0 GB, bytes/sec 3767278
2018/06/01 18:16:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 31m05s, bytes sent: 7.0 GB, bytes/sec 3773062

On second terminal I control memory usage with htop and RAM is constantly consumed and when all memory is used service crashes. After restart 0 bytes of snapshot is send:

++ hostname -f
+ dgraph server --my=dgraph-server-0.dgraph-server.default.svc.cluster.local:7080 --lru_mb 4096 --zero dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 18:18:59 gRPC server started.  Listening on port 9080
2018/06/01 18:18:59 HTTP server started.  Listening on port 8080
2018/06/01 18:18:59 worker.go:89: Worker listening at address: [::]:7080
2018/06/01 18:18:59 groups.go:78: Current Raft Id: 1
2018/06/01 18:18:59 pool.go:108: == CONNECT ==> Setting dgraph-zero-0.dgraph-zero.default.svc.cluster.local:5080
2018/06/01 18:18:59 groups.go:105: Connected to group zero. Assigned group: 0
2018/06/01 18:18:59 pool.go:108: == CONNECT ==> Setting dgraph-server-1.dgraph-server.default.svc.cluster.local:7080
2018/06/01 18:18:59 draft.go:170: Node ID: 1 with GroupID: 1
2018/06/01 18:18:59 node.go:228: Found hardstate: {Term:3 Vote:1 Commit:7 XXX_unrecognized:[]}
2018/06/01 18:18:59 node.go:240: Group 1 found 7 entries
2018/06/01 18:18:59 draft.go:926: Restarting node for group: 1
2018/06/01 18:18:59 raft.go:567: INFO: 1 became follower at term 3
2018/06/01 18:18:59 raft.go:315: INFO: newRaft 1 [peers: [], term: 3, commit: 7, applied: 0, lastindex: 7, lastterm: 3]
2018/06/01 18:18:59 node.go:118: Setting conf state to nodes:1 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"_predicate_" value_type:STRING list:true 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"element.type.in" value_type:UID directive:REVERSE 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"element.type.in" value_type:UID directive:REVERSE 
2018/06/01 18:18:59 mutation.go:181: Done schema update predicate:"_share_hash_" value_type:STRING directive:INDEX tokenizer:"exact" 
2018/06/01 18:19:03 raft.go:749: INFO: 1 is starting a new election at term 3
2018/06/01 18:19:03 raft.go:580: INFO: 1 became candidate at term 4
2018/06/01 18:19:03 raft.go:664: INFO: 1 received MsgVoteResp from 1 at term 4
2018/06/01 18:19:03 raft.go:621: INFO: 1 became leader at term 4
2018/06/01 18:19:03 node.go:301: INFO: raft.node: 1 elected leader 1 at term 4
2018/06/01 18:19:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 14s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:19:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 44s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 59s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:20:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 01m59s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:21:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 02m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m00s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:22:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 03m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m00s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m04s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m24s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:46 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m35s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:51 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m40s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:23:56 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m45s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:01 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m50s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:06 predicate.go:250: Sending SNAPSHOT: Time elapsed: 04m55s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:11 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m00s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:16 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m05s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:21 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m10s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:26 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m15s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:31 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m20s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:36 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m25s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:41 predicate.go:250: Sending SNAPSHOT: Time elapsed: 05m30s, bytes sent: 0 B, bytes/sec 0
2018/06/01 18:24:43 node.go:118: Setting conf state to nodes:1 nodes:2 

After that, server is sometimes returns correct results and sometimes return nothing.

• Expected behaviour and actual result.

Replication is finished and server returns correct results always

I setup dgraph using your docs - https://docs.dgraph.io/get-started

Then it says to connect to ratel to start running queries.

However, it won't connect at all. Everything is running fine according to docker logs.

There is some mention of a default password, but no mention of what that password is.

There are instructions on how to update the password using dgraph acl command but when I go into the console on the dgraph zero server and try to run that command, it says unknown command "acl" for "dgraph".

Are there any instructions for getting started that actually detail everything you need to do?

Why can't I set the default password via an environment variable similar to many other docker images?

Dgraph version:

Dgraph version   : v1.0.0
Commit SHA-1     : 7a64f54b
Commit timestamp : 2017-12-18 15:04:40 +0530
Branch           : HEAD

Dgraph linux binary: dgraph-linux-amd64-v1.0.0.tar.gz

OS: Manjaro (Based on Arch Linux. I consider it's the same as arch linux)

How do I run dgraph:

dgraph zero --port_offset -2000 -w ${db_path}/zw/ >> ${log_file_zero} 2>&1 &
sleep 10
dgraph server --memory_mb 1024 --zero localhost:5080 -p ${db_path}/p/  -w ${db_path}/w/ >> ${log_file_server} 2>&1 &

1. Create the schema by following https://tour.dgraph.io/moredata/1/
2. Load the dataset with dgraph live -r 1million.rdf.gz --zero localhost:5080
3. Shutdown and Restart dgraph zero and server to free up RAM.
4. Apply the schema mutation at https://tour.dgraph.io/search/1/

After the above 4 steps, the CPU usage is high about 40% busy when I don't do anything. I tried to shutdown and restart the dgraph zero and server many times. The CPU usage is always high when idle. It's not high all the time. It follows some kind of patterns. The CPU usage is high for several minutes, low for several minutes and then high again.

This never happens in pre v1.0.0 versions such as v0.9.4, v0.9.2 etc...

BTW, when I shutdown the dgraph server properly with curl http://localhost:8080/admin/shutdown, I always see the following snippet:

goroutine 511 [running]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).sendToWriteCh(0xc420444300, 0x0, 0x0, 0x0, 0x10, 0x10, 0xc43218e460)
    /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:609 +0x15f
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).batchSet(0xc420444300, 0x0, 0x0, 0x0, 0x0, 0x10)
    /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:679 +0x4d
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).purgeVersionsBelow(0xc420444300, 0xc43345d290, 0xc43218e460, 0x10, 0x10, 0x8, 0x0, 0x0)
    /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:931 +0x453
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*ManagedDB).PurgeVersionsBelow(0xc42014e138, 0xc43218e460, 0x10, 0x10, 0x8, 0x0, 0x0)
    /home/travis/gopath/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/managed_db.go:79 +0x127
github.com/dgraph-io/dgraph/posting.periodicPurgeOldVersions.func1()
    /home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:244 +0x2a2
github.com/dgraph-io/dgraph/posting.periodicPurgeOldVersions()
    /home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:254 +0x40
created by github.com/dgraph-io/dgraph/posting.Init
    /home/travis/gopath/src/github.com/dgraph-io/dgraph/posting/lists.go:211 +0xe1

I am not sure if this is related or not. This never appears before version 1.0.

One more werid part in version 1.0.0 is if I remove sleep 10 in the above script when I start dgraph, that means dgraph zero and server start at the same time. I can see about 9000 lines of:

2017/12/19 15:39:27 groups.go:673: Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader.

in the dgraph server log (stdout).

I don't think this is expected and never see such behavior before version 1.0. I am not sure if I should open another bug for this or not.

I attached three CPU profiles:

pprof.dgraph.samples.cpu.001.pb.gz

pprof.dgraph.samples.cpu.002.pb.gz

pprof.dgraph.samples.cpu.003.pb.gz

Have you tried Dgraph before this proposal? and did not find anything similar?

None

What you wanted to do.

I want to know when Dgraph is finished with a graphql schema drop operation so that I can safely insert a new schema without types being missing.

What you actually did.

I tried to insert a new schema after dropping the old schema through the GraphQL Admin API and waiting a short pause (5 seconds). The old schema and new schema share many of the same types. After inserting the new schema, several types (that are present in the new and old schema) no longer exist according to Dgraph

Why wasn't it great, with examples.

This is not great as it causes data insertion to fail for types that are missing.

Additional information.

Dgraph discuss post: https://discuss.dgraph.io/t/unknown-type-issue-after-dropping-dgraph-schema-data-and-then-resetting-the-schema/18108/3

Change: Added new automated tests for backup/restore (full as well as incremental ) on NFS and S3.

Addition:1]Included newly created TLS certificates for alpha's and zero's required for multiple HA cluster creation. 2]Added two new Docker files to create nfs-client+dgraph and nfs-server. 3] Added new targets to Makefile for creating docker images with nfs-client as well as docker image for NFS server

PS. This PR depends A LOT in the Roaring Bitmaps. ref: https://github.com/dgraph-io/dgraph/commit/1134839dec4c6c6d0ae2eb6b8db76f8b33129a24

This PR extends support for var inside the @groupby query on a scalar predicate at root. for example the given query now doesn't result to error:-

var(func: ... ) @groupby(age) {
  c as count(uid)
}

Now the uid variable c contains map of uid to count of uids with the same age. Suppose for the data :-

        "uid": "0x1",
        "age": 38
      },
      {
        "uid": "0x17",
        "age": 15
      },
      {
        "uid": "0x18",
        "age": 15
      },
      {
        "uid": "0x19",
        "age": 17
      }

The following groupby query:-

var(func: uid(1, 23, 24, 25)) @groupby(age) {
      c as count(uid)
}
 me(func: uid(c)) {
      uid
      val(c)
}

returns the result:-

"data": {
    "me": [
      {
        "uid": "0x1",
        "val(c)": 1
      },
      {
        "uid": "0x17",
        "val(c)": 2
      },
      {
        "uid": "0x18",
        "val(c)": 2
      },
      {
        "uid": "0x19",
        "val(c)": 1
      }
    ]
  }

The behaviour will be the same if there are multiple predicates and some of them are uid predicates. However, it is only supported at root currently. Hence the given query will still return error:-

var(func: .....) {
   friend @groupby(age) {
       a as count(uid)
   }
}

What you wanted to do.

v21.12 PR https://github.com/dgraph-io/dgraph/pull/7663

See Discuss https://discuss.dgraph.io/t/rfc-allow-language-tag-support-in-graphql/13027