Software development and release processes continues to improve to deliver value to the users faster and better to support business growth and relevance in this competitive market. To achieve this we focus on automating the path to production and any people or process related obstacles of a software on its way to the user. Generally, some of the goals of the golden paths, are:
- Remove interdependency and promote self service and provider & consumer relationship.
- Shift left - from people & process to technology & automation.
- Treat Platform-as-product and provide PaaS
- Secured and standardised by design
In this post, I will describe how Crossplane and ServiceBinding can help achieve these goals in the context of applications development and delivery and their consumption of external resources / services in the process.
Table of content
What are Crossplane and ServiceBinding
Why do we need Crossplane and ServiceBinding
How do we implement Crossplane and ServiceBinding
What are Crossplane and ServiceBinding
Crossplane:
Crossplane connects K8s cluster to external non-K8s resources as well as in cluster K8s resources and allows platform teams to build custom K8s APIs to consume those resources. Read more about it here: crossplane.io. Below are my highlights:
- It Enables self-service capability and creates segregation among dev, ops and security. This removes interdependencies on people and process and shifts responsibilities to left.
- It promotes platform-as-product principle where platform teams can combine external resources and simplify or customize the APIs presented to the platform consumers, Consumers experience true PaaS functionalities. This helps create better provider-consumer relationship.
- It promotes the principle of cloud-native development by abstracting away the cloud and vendor specific configurations that enriches consumers' self-serve experience. This is an immense value add in the context micro-services. For example: A Kind: PGDB represents a PgSql database to the consumers. The providers independently decide whether PostgreSQL is RDS or AzureDB or in-cluster pgsql operator. This is promotes cloud neutrality and flexibility to place the resource anywhere.
- In my opinion it is better than traditional IaaC (eg: Terraform). It is a native K8s construct/object, meaning I can deploy K8s object to create and manage these resources. This is super powerful as it can be done via gitops, reconciliation and K8s events for resources/infrastructures.
Service Binding:
It is a Kubernetes-wide specification for communicating services/resources secrets to workloads in an automated way. Read more about here: servicebinding.io. Below are my highlights:
- Provides an abstraction layer that simplifies the consumption of different and dynamic external/internal services/resources. This keeps deployment of applications simple and consistent across the portfolio.
- Simplifies the applications and services lifecycle by abstracting and decoupling the specifics, making it easier to manage and update services/resources without impacting the applications consuming them and vice versa.
- It enforces standardisation by design. Meaning, it defines and dictates how services are consumed in applications. Thus, providing consistency across application portfolio. This adds immense value in micro-services landscape where achieving consistency is critical to success.
- It is secured-by-design. It, almost entirely, removes the need to handle (supply, create etc) credentials and sensitive uri manually. Thus securing the application's development and release processes by reducing credential stuffing probability. .
Crossplane + ServiceBinding combo:
Here are some of the things (among many) we get when we use these 2 tools together in the right way:
- Crossplane handles the resources creation and management. It also abstracts the specifics of providers or vendors. App Owners need this type of capabilities to provision the resources through self-serve and Platform Ops need this to provide the resources ready of claiming like PaaS.
- ServiceBinding abstracts the vendor/provider specific details between an application and a resource. By surfacing the provisioned resource as a service, ServiceBinding simplifies and provides automated way to consume the resources. Simply put, App Owners and App Developers need this functionality to connect their applications to the resources in a consistent and standard way.
- Crossplane and ServiceBinding in combo provides a true end-to-end consumer and provider experience.
Why do we need Crossplane and ServiceBinding:
I will explain why Crossplane and ServiceBinding is a powerful combo by describing a scenario of an application's lifecycle. Here are some background info about the application (BTW, this is a real life scenario):
- The application is comprised of approx. 15 micro-services. These micro-services are created and managed by different teams and released and lifecycled at different pace. As the business grows there are probability that the application's micro-services will also increase in numbers.
- Most micro-services (of this application) will connect to resources for storing and retrieving data (eg: PostgreSQL, MongoDB, Redis, Kafka, SQS, In house APIs etc).
- The release path of micro-services consists of 4 environments: Dev, UAT, Stagein, Prod. Dev and UAT environments must utilise low cost resources (eg: local in-cluster pgsql) and they are recycled every quarter. However, the Stagein environments utilise resources that are Prod like (eg: RDS). Note: Prod environments are always a special case and too sensitive topic to impose my opinion on. So, I will leave prod environments to your thought and your business.
Lets visualise a scenario of 1 micro-service (connecting to 1 pgsql) without a provider-consumer model (I am sure this nightmare is familiar, but I would still like highlight again to put things into perspective).
- At the start of developing this micro-service, the app owner requests for a pgsql db for dev environment by raising a BAU ticket (or worst, via email).
- The IT Ops team reacts to that request and raises a SR to be approved by the platform/security team.
- The platform/security team reviews the SR and if it is deemed to satisfy the parameters they approves the request.
- Upon receiving the approval the IT Ops team creates the pgsql db in local K8s cluster (eg: using a pgsql operator like cloudnative-pg).
- The ops team, then, records the credential in a credential management system and supplies the app owner with other details like uri, username etc over email or walking to his/her desk or similar.
- The app operator uses these details and (assuming has access to the credential management system) gets the password from the central credential storage system and creates a configmap holding those details. He also modifies the deployment definition to mount the configmaps as volumes and instructs the developers how to read it.
- The developers adjusts the library and source code to reads as per instruction to connect to the pgsql db. Note: the structure of the connection details may vary for different apps and / or different resources or even same type of resource provisioned by different teams.
- The same process repeats for other environments eg: uat, stagein and prod. But, it varies for different db providers, eg: cloudnative-pg vs AWS RDS. During secret rotation this process (1-6) is repeated too. Multiply this process for 19 other micro-services. And this is just for 1 application.
- credential stuffing is a big risk.
- Turn around time is a big drawback. It may take anywhere between a day to multiple weeks.
- People and process interdependency is another drawback. Meaning, the business needs to accept increasing OPEX cost in the IT Ops department to handle influx of these type of request.
- Even if this is some how scripted the script itself is a tech-debt and it will become an overhead (eg: people shifting to other teams, API changes etc).
- There's no cohesiveness here. In a world of micro-services this model becomes chokepoint and fails to deliver on time and costs the business unnecessary spending and delay, thus negatively impacting the end-user.
- There's no provider-consumer relationship here. There's no platform-as-product principle here. So, the development and operation experience is negative and unsustainable.
I could accept this for monolith applications back in the day and but not for modern applications.
Now, lets see how we can solve these issues with a provider-consumer model implemented using Crossplane and ServiceBinding.
- With Crossplane available to the Platform Ops team, they create capability in the platform (K8s based environment) to provide the external resources (eg: pgsql instances) as services (PaaS). The security team also performs their review of the PaaS capabilities upfront. Thus it turns from reactive to proactive.
- With Crossplane available to the App Owners they self-serve themselves to create pgsql db in accordance to the environment (eg: local instance for dev and uat, rds instance for stagein) for the micro-services without having to depend on the other teams or raise a SR ticket.
They do not experience any difference in claiming the pgsql db in different environments (eg: in-cluster or in-the-cloud pgsql for dev, uat, stagein environments).
Using ServiceBinding they can also bind/integrate the provisioned resources with applications consistently across all environments without having to modify a single line in the deployment definition. This is auto handled by ServiceBinding.
The App Developers can also follow a standard way to connect to the resources for all the micro-services of the application using standard libraries for ServiceBinding (ie: Well-Known Secret Entries).
Some of benefits from the above worth highlighting here are:
- Better provider-consumer relationship leads to
- self-service,
- significantly reduces interdependencies,
- cohesive processes by shifting responsibility to the left,
- rich development and operation experience
- acceleration of the release cycles (true agile).
- This model is secured by design as the connection details and specifics are obfuscated and does not require manual handling. The risk of credential stuffing the almost eliminated.
- It also lets platform and app operators treat the platform-as-product. The need for BAU tasks are significantly reduced and the teams can focus on the core objectives such as platform improvement, application development etc. Worth mentioning that PaaS is a by product here.
- It decouples applications and app owners from underlying implementation of the resources. For example the in-cluster cloudnative-pg operator based pgsql vs in-the-cloud rds based pgsql will appear as just pgsql to app owners. This gives flexibility to ops team to swap underlying resource provider, rotate secrets without impacting developers or app owners.
So, the question here isn't why Crossplane + ServiceBinding.
Rather it's "why not"?
How do we implement Crossplane and ServiceBinding
Now that I have covered (and hopefully convinced) why Crossplane and ServiceBinding are such a powerful combo lets move on the how to implement part. And it is surprisingly easy. Unfortunately, in my opinion, the documentations makes it seem hard. See below how I implemented a pgsql-as-a-service for my application in dev, uat and stagein environment.
Lets consider this diagram for our implementation:
First, I will create Crossplane for in-cluster pgsql db as service. We will use CloudNative-PG for it. Assume the K8s cluster already has the CloudNative-PG operator installed on it. The objectives here are:
- The DB instances should be in a separate namespace so that we can limit the db server space to only a handful of people (eg: ops team).
- Apps team will need the connection details in the apps' namespaces.
- Publish the connection details according to Well-known Secret Entries.
Note: The above are also the reasons why we cannot use CloudNative-PG directly and need to manipulate using Crossplane to meet our usecase.
Since CloudNative-PG runs in K8s in-cluster, we will need a K8s provider:
# Make sure provider-kubernetes has enough permissions to install your objects into cluster
#
# You can give admin permissions by running:
# SA=$(kubectl -n crossplane-system get sa -o name | grep provider-kubernetes | sed -e 's|serviceaccount\/|crossplane-system:|g')
# kubectl create clusterrolebinding provider-kubernetes-admin-binding --clusterrole cluster-admin --serviceaccount="${SA}"
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-kubernetes
spec:
package: "crossplanecontrib/provider-kubernetes:main"
---
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
name: default
spec:
credentials:
source: InjectedIdentity
Composition to create CloudNative-PG instances:
--- apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: xcnpginstances.dev.pgdb.local spec: compositeTypeRef: apiVersion: pgdb.local/v1alpha1 kind: XCNPGDevInstance publishConnectionDetailsWithStoreConfigRef: name: default resources: - base: apiVersion: kubernetes.crossplane.io/v1alpha1 kind: Object spec: forProvider: manifest: apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: PlaceHolder namespace: dev-instances labels: type: postgresql provider: cloudnativepg spec: instances: 1 storage: size: 1G extra: type: postgresql provider: cloudnativepg connectionDetails: - apiVersion: v1 kind: Secret namespace: dev-instances fieldPath: data.host toConnectionSecretKey: host - apiVersion: v1 kind: Secret namespace: dev-instances fieldPath: data.port toConnectionSecretKey: port - apiVersion: v1 kind: Secret namespace: dev-instances fieldPath: data.username toConnectionSecretKey: username - apiVersion: v1 kind: Secret namespace: dev-instances fieldPath: data.password toConnectionSecretKey: password - apiVersion: v1 kind: Secret namespace: dev-instances fieldPath: data.dbname toConnectionSecretKey: database - apiVersion: v1 kind: Secret namespace: dev-instances fieldPath: data.uri toConnectionSecretKey: uri - apiVersion: postgresql.cnpg.io/v1 kind: Cluster namespace: dev-instances fieldPath: metadata.labels['type'] toConnectionSecretKey: type - apiVersion: postgresql.cnpg.io/v1 kind: Cluster namespace: dev-instances fieldPath: metadata.labels['provider'] toConnectionSecretKey: provider writeConnectionSecretToRef: namespace: dev-instances connectionDetails: - fromConnectionSecretKey: host - fromConnectionSecretKey: port - fromConnectionSecretKey: username - fromConnectionSecretKey: password - fromConnectionSecretKey: database - fromConnectionSecretKey: uri - name: provider value: "cloudnativepg" - name: type value: "postgresql" patches: - fromFieldPath: metadata.name toFieldPath: spec.forProvider.manifest.metadata.name type: FromCompositeFieldPath # - fromFieldPath: spec.instancesCount # toFieldPath: spec.forProvider.manifest.spec.instances # type: FromCompositeFieldPath - fromFieldPath: spec.storageGB toFieldPath: spec.forProvider.manifest.spec.storage.size transforms: - string: fmt: '%dG' type: Format type: string type: FromCompositeFieldPath # important: this will place the secret in app namespace. # This is important because, this is how: # - we create the pg instance in a designated namespace. in this declaration it is hardcoded as: dev-instances # - we create the connection secret in the application's namespace. in this declaration the value will come from the XRD. # From there we can use ServiceBinding to bind this secert in the app's NS for connection and The DB will exist in the its designated NS # This is so that, provider <-> maintainer <-> consumer relationship remains intact with self serve capability. - fromFieldPath: spec.appNamespace toFieldPath: spec.writeConnectionSecretToRef.namespace type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.writeConnectionSecretToRef.name transforms: - string: fmt: '%s-cnpg-secret' type: Format type: string type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[0].name transforms: - string: fmt: '%s-app' # Note: matching the name of the secret auto generated by CloudNative-PG type: Format type: string type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[1].name transforms: - string: fmt: '%s-app' type: Format type: string type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[2].name transforms: - string: fmt: '%s-app' type: Format type: string type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[3].name transforms: - string: fmt: '%s-app' type: Format type: string type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[4].name transforms: - string: fmt: '%s-app' type: Format type: string type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[5].name transforms: - string: fmt: '%s-app' type: Format type: string type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[6].name type: FromCompositeFieldPath - fromFieldPath: metadata.name toFieldPath: spec.connectionDetails[7].name type: FromCompositeFieldPath readinessChecks: - type: MatchString fieldPath: status.atProvider.manifest.status.conditions[0].reason matchString: "ClusterIsReady"
To expose the capability to the App owner we will also need to create a XRD matching with spec.compositeTypeRef.
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xcnpgdevinstances.pgdb.local
spec:
connectionSecretKeys:
- provider
- type
- database
- host
- port
- username
- password
- uri
group: pgdb.local
names:
kind: XCNPGDevInstance
plural: xcnpgdevinstances
claimNames:
kind: CNPGDevInstance
plural: cnpgdevinstances
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
appNamespace:
type: string
storageGB:
type: integer
required:
- appNamespaceThat's it. Now the app owner has the self-serve ability to create in cluster pgsql instances for their applications using Crossplane claim object.
Here's an example of a claim:
---
apiVersion: pgdb.local/v1alpha1
kind: CNPGDevInstance
metadata:
name: account-db
namespace: dev-instances
spec:
appNamespace: my-customer-portal
storageGB: 1See more details about XRD, Composition, Claim etc see the Crossplane official documentation.
Similarly for AWS RDS pgsql we will below crossplane object:
- AWS provider family and RDS provider
- Composition for RDS
- XRD for RDS -- here we can restrict which parameters will be allowed for creating a RDS instance. For example: we can restrict it to create only pgsql (and not mysql).
See the example here.
The claim object for RDS will look like:
---
apiVersion: pgdb.rds/v1alpha1
kind: RDSPGDBInstance
metadata:
name: account-db
namespace: stagein-instances
spec: region: us-west-2
size: small
dbName: account-dbNow that the self-service capabilities to create and manage pgsql db is created and psql is provided as PostgreSQL-as-Service expand the same diagram with ServiceBinding to consume the pgsql db into the application:
The objectives here are:
- Provide App Developers ability to bind "Database" and abstract away the specifics.
- Create a loose couple between the connection details (eg: K8s secret object containing connection information) and the application. This is so that the secret rotation or db cycling can happen without impacting the application or the development process (eg: not requiring the developer to bind again).
To achieve the above as an ops team we will create a K8s CRD:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.example.sb
spec:
group: example.sb
names:
kind: Database
plural: databases
scope: Namespaced
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
type: object
properties:
provider:
type: string
database:
type: string
server:
type: string
status:
type: object
properties:
binding:
type: object
properties:
name:
type: string
This gives the app owners the self-service capability to consume db as a service. Below is an example:
apiVersion: example.sb/v1alpha1
kind: Database
metadata:
name: account-db-service
namespace: test
spec:
provider: cloudnative-pg # optional
database: account-db # optional
server: cnpg # optional. Possible values are: cnpg, rds
status:
binding:
name: account-db-hg2d9-cnpg-secret # required. This is the secret created from XP Claim.Once the Database is available for binding to the application.
Here's a sample application's deployment declaration with ServiceBinding:
---apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- env:
- name: spring_profiles_active
value: pgsql
image: my.registry.io/workload/my-app@sha256:140ca10d7c79xxx
...
serviceAccountName: default
---apiVersion: servicebinding.io/v1beta1
kind: ServiceBinding
metadata:
name: account-db-binding
namespace: test
spec:
service:
apiVersion: example.sb/v1alpha1
kind: Database
name: account-db-service # Required. Reference to the Database object.
workload:
apiVersion: apps/v1
kind: Deployment
name: my-app # Required. Reference to the application's deployment object.More details (and more flexibility to bind) please the official ServiceBinding documentation.
Here, the deployment declaration remains intact (as-is or as-was). We are adding a ServiceBinding declaration which auto injects the secret into the workload as volume mount. No manual mount declaration required.
For app owners and developers this means that the deployment does not require reading from configmap or mounting volume AND on top, it provides a consistent way to read/connect in the application source code.
For example: In a Java SpringBoot app (using Spring Cloud Binding) connecting pgsql the application.yml (or application.properties) will look like below:
spring:
config.activate.on-profile: pgsql
datasource:
driverClassName: org.postgresql.Driver # No need for any of the below. The Spring Cloud Data library handles it automatically.
# username: ph
# password: ph
# url: ph
jpa:
hibernate.ddl-auto: none #create-drop #none #validate
database-platform: org.hibernate.dialect.PostgreSQLDialect
sql.init:
mode: always #never #always
schema-locations: classpath:schema-pgsql.sql
data-locations: classpath:data-pgsql.sqlSee more details about the standard libraries (eg: Spring Cloud Data) that handles ServiceBindings in the official ServiceBinding.io documentation: here.
That's it.
F.A.Q:
Q: We use external secret store (eg: vault) for storing connection details for security reason. Do we still use ServiceBinding?
A: Yes. ServiceBinding can bind directly to secret.
Q: The secret generated by Crossplane (as shown in this post) is plain K8s secret which is not secured. So, how is this "secured by design"?
A: Firstly, I think, plain K8s secrets are fine. See why in this post here. Please read it and really ponder whether maintaining appropriate methods can eliminate the need for external secret manager (and cost savings in the process). Below are some hints:
- the K8s secrets are NOT handled by human and
- they are exposed to as needed basis and
- they are rotated without any impact
So, if we implement the above (and we really should) what are the chances of credential stuffing. I recommend doing a threat analysis on it before making a blanket decision that we cannot use plain K8s secret for "security" reason.
Secondly, (if security still remains a concern then) Crossplane has integration to external secret manager. It's probably overkill, in my opinion, but the requirement can be fulfulled. See the documentation here.
Q: This sounds ok. But these are 2 small pieces of the puzzle. What about the rest? What about enterprise grade product? What about CICD? What about GUI?
A: Firstly, Yes, agreed. Crossplane and ServiceBinding are 2 pieces of the puzzle of an end-to-end app, dev and devops platform (not small though). If there's preference and budget, in the business, to get a enterprise grade product then you may want to look at a product called Tanzu Application Platform. It truly covers end-to-end and is based of off open source tools; Crossplane and ServiceBinding are 2 among several others.
Secondly, you may have noticed that there is a small disjoint in the app owners experience. That is:
- Before deploying Service Binding as a part of app deployment an app owner needs to create the "Service" object (from the CRD) which points to an auto generated K8s secret. This means that the app owner must discover the auto generated secret (not handle it, just discover). Although this issue can be easily eliminated through a mutually agreed naming convention but it is not forced by design.
- The goal was to completely remove the need to worry about secrets. Although this is categorised as just a discovery and not exactly handling, this is still one more not-ideal thing.
An enterprise grade end-to-end platform, such as Tanzu Application Platform, removes this type of disjoint and represents Crossplane and ServiceBinding in integrated manners. See below diagram:
By using Tanzu Application Platform (aka TAP) we can provide a more integrated consumer experience as well as eliminate the need for any touchpoint with any sort of secret (even discovering).
Moreover, it simplifies the self-service experience to a greater extent. In this context it looks like this:
- ClassClaim to create the external resource
- WorkloadDefinition to create the workload to run
The rest are auto created / bounded as defined or referenced in the workload definition. In this case referring to the ClassClaim will auto create ServiceBinding to the resource. Similarly, it also does several other things such as generate yamls for deployment, API registration, Service Mesh objects, Environment Vars etc if instructed in the WorkloadDefinition. The Workload Definition is one single definition of a workload and all it consumes. Users (app owners, plarform owners) do not require any additional CRDs.
Crossplane and ServiceBinding are 2 components among several others in Tanzu Application Platform (eg: CI, CD, KNative-Serverless, API Gateway, GUI, Backstage etc). So, in my opinion, for an end-to-end dev, devops and platform ops platform TAP is well worth the license cost.
Conclusion:
Modern app development and delivery simply does not work with legacy IT driven ops model. We need modern tools/techs and ops model (eg: devops) for it. The usecases and benefits, of Crossplane and ServiceBinding and the DevSecOps model levering them, that are highlighted in this post are supporting evidence of that.
Let's not waste time trying to fit in modern app development into the IT Ops tooling and processes which cause burnout and failure by imposing deadlines to deliver. Rather invest in the right tools and technologies, shift left, create PaaS and the acceleration of application development and delivery will follow and so will the benefits to the business.
Comments
Post a Comment