Resolving Istio Revision Webhooks Misconfiguration

Resolving Istio Revision Webhooks Misconfiguration

Our development team alerted us to an issue where their virtual service changes were not being applied. Upon investigation, we discovered that the Istio control plane was malfunctioning. Logs from the Istio Ingress gateway revealed exceptions caused by invalidly configured resources, preventing new changes from being applied.

Root Cause Analysis

Further analysis revealed that some virtual services were misconfigured. Specifically, while multiple routes were defined in the virtual services, no weights were assigned to these routes. This misconfiguration caused exceptions in the Istio control plane, leading to a “blackhole” scenario where no changes could be processed.

The critical question was: how did these misconfigured resources bypass validation? Istio’s validation webhook is designed to prevent such issues. However, we found that the webhook was misconfigured. It was pointing to an object selector that did not match any virtual services, effectively bypassing validation and allowing misconfigured resources to be created or updated.

Immediate Actions

Team Communication

We promptly informed the development team about the issue and advised them to refrain from making changes to virtual services until the problem was resolved. Additionally, we emphasized the importance of proper configuration and validation to prevent similar issues in the future.

Correcting the Webhook Configuration

We collaborated with the container team to review the Istio installation process. The team identified and corrected the webhook configurations, ensuring they pointed to the correct object selector matching virtual services. After implementing these changes, we tested the validation webhook by attempting to create a misconfigured virtual service. The webhook successfully blocked the creation, confirming that it was functioning as intended.

Bug or Feature?

The existence of such a critical misconfiguration in the Istio installation process raised questions: was this a bug or an intentional feature? A review of GitHub issues revealed that this problem had been reported multiple times. However, the responses suggested workarounds, such as creating a custom webhook or removing object selectors, which require additional configuration and maintenance.

We believe this issue should be addressed in the default Istio installation process to ensure proper validation out of the box. After further investigation, we discovered a permanent solution: adding the flag --set values.defaultRevision=default during installation. This flag creates an additional validating webhook, istiod-default-validator, alongside the existing istiod-validator-istio-system webhook.

Understanding the Difference

• istiod-validator-istio-system: This webhook includes the object selector istio.io/rev=default, meaning it only validates resources labeled with istio.io/rev=default.
• istiod-default-validator: This webhook does not include any object selector, allowing it to validate all resources, regardless of their labels.

By adding the --set values.defaultRevision=default flag, we ensured the creation of the istiod-default-validator, which validates all resources and prevents misconfigurations.

This configuration is documented in the Istio installation guide. While we may have overlooked it during the initial setup, we believe the default installation should include both webhooks to ensure robust validation.

Conclusion

In summary, we resolved the Istio control plane issue by correcting the webhook configurations and ensuring the validation webhook operated correctly. We also highlighted the importance of proper configuration and validation to maintain the stability and reliability of the Istio control plane. Finally, we implemented a permanent solution by adding the --set values.defaultRevision=default flag during installation, creating a validating webhook that prevents misconfigurations. This approach ensures the Istio control plane functions correctly, allowing changes to be applied seamlessly.

---

Special thanks to Batuhan Apaydın for their valuable contributions.