Problem consuming from stream when moving from 1 shard to 2 shards.

[jackmitchelwidman]

We are using reactive-kinesis for the time to consume from a stream. With 1 shard, everything works fine. When we bump it up to 2 shards, the consumer shuts down soon after it starts. The log error is:

2018-05-09 12:03:48,991 [warn] c.w.r.k.c.ConsumerWorker - Worker for shard Some(shardId-000000000002): Skipped checkpointing on shutdown 2018-05-09 12:03:48,993 [info] c.w.r.k.s.KinesisSourceGraph - Consumer shutdown for shard shardId-000000000002 2018-05-09 12:03:48,995 [error] c.a.s.k.c.l.w.ShutdownTask - Application exception. java.lang.IllegalArgumentException: Application didn't checkpoint at end of shard shardId-000000000002 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask.call(ShutdownTask.java:110)

Is there any configuration that must be done on the consumer? Is there something we have to do in code? Thanks for your time.

[etspaceman]

Duplicate of #37?

[jackmitchelwidman]

Is this a known issue? We are on a short deadline and we need to know if we can use this library in production where there are more than 1 shard. @markglh?

[markglh]

Trying to reproduce this earlier but I couldn't. We regularly use many shards without issues - are you able to reproduce this with an integration test? I'll try again tomorrow

[jackmitchelwidman]

We believe its because we switched from 1 to 2 shards. We consumed for some time with 1 before moving to two. This apparently closes the first shard and creates two new ones. Perhaps the closed shards have something to do with the problem? At any rate, it seems the the problem may lie in the changing of the number of shards.

[etspaceman]

@jackmitchelwidman Yeah that's exactly what I had determined in #37. Scaling the kinesis shards yields errors while the old shard is "closed" but still physically available as a part of the kinesis stream.

There's a linked AWS thread with some information on what they recommend on #37.

[jackmitchelwidman]

@etspaceman As a temporary measure, we deleted the stream and recreated it with two shards, one ending in 0 and one ending in 1. When the consumer starts up, we get an error about a lease still existing for shard 2 (which I assumed got deleted when we deleted the stream). Have you seen this behavior?

[etspaceman]

Sorry for the late reply @jackmitchelwidman - I have not seen that behavior personally so I'm not of much help here :(

[veshnu]

Are there any fix planned?