A119: Auto-Sharding LB Policy - #551
Conversation
|
@markdroth @ejona86 @dfawley |
|
I'm going to be making some structural changes to the spec based on our review today. Will ping here once it is ready for review. @markdroth @ejona86 @dfawley @shivaspeaks @pawbhard Please expect some changes to the spec, but not wholesale changes. So, you could start looking at it from an implementation pov. Thanks. |
|
The PR is ready to be looked at. |
markdroth
left a comment
markdroth
left a comment
There was a problem hiding this comment.
This looks really good!
I have a lot of comments here, but they're mostly fine details or clarifications -- the high-level shape of this is solid, and the doc does a good job of describing it.
Please let me know if you have any questions. Thanks!
markdroth
left a comment
markdroth
left a comment
There was a problem hiding this comment.
This is getting closer!
Please let me know if you have any questions. Thanks!
| // A timeout value for fallback to kick in when no assignments have been | ||
| // received from the sharding service. | ||
| // Defaults to 60 seconds if not specified. | ||
| google.protobuf.Duration fallback_timeout = 6; |
There was a problem hiding this comment.
It doesn't really matter whether it's defined here or in A121. I guess it should be in whichever of the two gRFCs is going to be merged last.
Please coordinate with @mbissa.
| allEndpointsInSlice []int // Indices into sliceMap.allEndpoints | ||
| endpointsByState [5][]int // Array indexed directly by connectivity.State (ranges 0..4) |
There was a problem hiding this comment.
Okay. @pawbhard, let's consider this for the C-core implementation.
| 1. Parsing a `GrpcService` proto embedded within an LB policy's configuration | ||
| into its internal representation, requires access to the following: | ||
| * the complete bootstrap configuration to access the `allowed_grpc_services` | ||
| section of the bootstrap configuration. | ||
| * configuration of the specific xDS server that delivered this resource, to | ||
| determine if the server is to be trusted or not. |
|
@markdroth : I think I've addressed all your comments. Thanks for the detailed review. |
| If validation fails, the LB policy must terminate the stream to the sharding | ||
| service, and attempt to re-establish it. |
There was a problem hiding this comment.
Is this really the right thing to do? If we get an invalid assignment, that's likely a bug in the slicer service, and it's not clear to me that reestablishing the stream will fix it -- we'll probably just get the same bad data again after we restart the stream. But restarting the stream will cause load on the server, so I'd be worried about a cascading failure here: a bug in the slicer service could cause all of the clients to start terminating and reconnecting streams, which could cause a lot of unexpected load.
I suggest asking the slicer folks how they handle this kind of case today.
There was a problem hiding this comment.
I've updated this section. PTAL. Thanks.
There was a problem hiding this comment.
Also, updated to include a new component named AssignmentProvider which will encapsulate all details of the interaction with the sharding service and will provide usable assignments to the LB policy.
One thing I'm still a little uncertain is whether if one of those validations fail (like one slice entry contains an invalid index), do we still use the new assignment but send an AssignmentAck with accepted set to true but with the error message populated? (I currently have it the other way around, but willing to change it).
If we do the above, then the only cases where we would end up setting the accepted field to false would be when we have no usable slices in the assignment. Is that your understanding too?
Thanks
| If a new gRPC channel is created: | ||
| * Create a new `Shard` stream on the newly created gRPC channel, and, | ||
| * Close the previously created gRPC channel to the sharding service | ||
| * If the `slicing_target` field has changed, create a new `Shard` stream because |
There was a problem hiding this comment.
In this case do we discard existing and enter assignment pending ?
There was a problem hiding this comment.
No, we start a new initial assignment timer but keep using the previous assignment until we either get a new assignment or the timer fires. This is covered in the "Fallback at Startup" section above.
5 participants
No description provided.