A cloud-native, event-driven architecture that cleverly leverages cloud resources and events as the primary communication mechanism between services. This architecture makes the system scalable, resilient, and agile by separating the components, enabling real-time response to events. It uses event streaming and serverless computing for distributed event-driven communication and processing. Oracle Cloud Infrastructure Events is a powerful automation tool that creates automated actions based on changes in the state of various cloud resources. It allows us to create rules that deliver events to specific OCI resources, such as Oracle Cloud Infrastructure Flows**, OCI Functions, and Oracle Cloud Infrastructure Notifications. In some cases, we need to publish a message to a private stream. While OCI events can deliver events directly to the public stream endpoint, it does not currently support direct message delivery to the private stream endpoint because private endpoints can only resolve within the same virtual cloud network, while OCI events cannot push messages to private streams. However, this can be achieved by using the OCI function as the action type in the event rules. This reference architecture provides a comprehensive overview of how to leverage OCI functions and OCI events to deliver messages to private stream endpoints. In this reference architecture, we show how to leverage OCI functions and OCI events to publish messages to a private OSS streaming endpoint when a file in Oracle Cloud Infrastructure Object Storage is created or updated. The architecture diagram clearly depicts the process of data flow: it starts with uploading files to a specific bucket in OCI object storage. Subsequently, based on the defined event rule conditions, the OCI event is triggered. The emitted event then calls a function that extracts the data from the uploaded file and publishes the data and event messages to the private stream endpoint configured in the OCI function. It is important to note that in order to ensure that OCI functions can access the private streaming message endpoints, the streaming service and function should be created in the same virtual cloud network (VCN) and private subnet. The following diagram illustrates this reference architecture in detail. 
First, you'll need to set up your OCI function app and the OCI Streams service (Stream Pools and Streams).
Please prepare the following information. These values are required to configure the environment variables of the OCI function:
User IDUser OCID used for authentication.
Tenant IDTenant OCID. It can be found in the user profile.
FingerprintsUsed to authenticate OCI APIs.
areaCreate the requested region identifier.
apiPrivate keyAPI private key file location.
Stream ocidOSS stream OCID.
Authentication tokensAuthentication token in the user settings.
Stream endpointsOSS stream endpoint (obtained from Message Endpoint on the flow information screen)
The schema consists of the following components:TenantsWhen you sign up for Oracle Cloud Infrastructure, Oracle establishes a secure and separate part of the cloud for you: your tenant. You can create, organize, and manage your resources in Oracle Cloud within your tenant. Tenant can be considered synonymous with a company or organization. Typically, a company will have one tenant and its organizational structure will be reflected within that tenant. A single tenant is typically associated with a single subscription, which typically has only one tenant. areaAn Oracle Cloud Infrastructure Region is a geographic region that contains one or more data centers, called Availability Domains. Regions are independent of each other and can cross countries or even continents. PartitioningPartitioning is a cross-region logical partition within an Oracle Cloud Infrastructure tenant. You can use partitions to organize resources in Oracle Cloud, control access to those resources, and set usage quotas. In order to control access to resources in a specific partition, you need to define policies that specify who can access those resources and what actions they can perform. Availability domainsAvailability domains are independent, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, providing failure tolerance. Availability domains don't share infrastructure, such as power or cooling systems, or internal availability domain networks. As a result, the failure of one availability domain is unlikely to affect other availability domains within that region. Fault domainsA fault domain is a grouping of hardware and infrastructure within an availability domain. Each availability domain contains three fault domains for independent power and hardware. When you distribute resources across multiple fault domains, your application can withstand physical server failures, system maintenance, and power failures within the fault domains. Virtual Cloud Networks (VCNs) and SubnetsA virtual cloud network is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like a traditional data center network, a VCN gives you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks, which you can change after the VCN is created. You can divide your VCN into subnets, which can be scoped in a region or availability domain. Each subnet consists of contiguous address ranges that do not overlap with other subnets in the VCN. You can change the size of the subnet after it has been created. Subnets can be public or private. Safe listsFor each subnet, you can create security rules that specify the traffic, destination, and type of traffic that must be allowed in and out of the subnet. Services GatewayThe Services Gateway provides access from the VCN to other services, such as Oracle Cloud Infrastructure Object Storage. Traffic from the VCN to the Oracle service travels through the Oracle network infrastructure and does not traverse the Internet. Object storageObject storage provides fast access to large amounts of structured and unstructured data, including database backups, analytics data, and rich content such as images and data. You can store your data securely and securely and then retrieve it directly from within the internet or cloud platform. You can seamlessly expand your storage space without compromising performance or service reliability. For "hot" storage that requires fast, instant, and frequent access, use standard storage; For "cold" storage that is kept for a long time and accessed little or no time, use archive storage. EventsEvents generated by Oracle Cloud Infrastructure services are structured messages that describe changes in resources. These events cover create, read, update, or delete (CRUD) operations, resource lifecycle state changes, and system events that affect cloud resources. StreamsOracle Cloud Infrastructure Flows** provides a fully managed, scalable, and durable storage solution for ingesting continuous, massive streams of data that you can consume and process in real time. Streaming can be used to ingest large amounts of data, such as application logs, operational telemetry, clickstream data; or for other use cases of a publish-subscribe messaging model that produces and processes data continuously and sequentially. functionsOracle Cloud Infrastructure Functions is a fully managed, multi-tenant, highly scalable, on-demand Function-as-a-Service (FaaS) platform. It is powered by the FN project open-source engine. Functions allow you to deploy ** and call it directly or trigger it in response to an event. Oracle functions use docker containers hosted in the Oracle Cloud Infrastructure registry. LogsLogs is a highly scalable and fully managed service that provides the following types of cloud resource log access: Audit logs: Logs related to events emitted by the audit service.
Service logs: logs emitted by individual services, such as API gateways, events, functions, load balancing, object storage, and VCN flow logs.
Custom logs: Logs that contain diagnostic information from custom applications, other cloud providers, or on-premises environments.
RecommendedUse the following recommendations as a starting point for implementing this reference schema using OCI functions and OCI events. Your needs may differ from the architecture described here. vcnWhen you create a VCN, determine the number of CIDR blocks you need and the size of each block based on the number of resources you plan to connect to in the VCN subnet. CIDR blocks within a standard private IP address space are used.
Choose a CIDR block that does not overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or other cloud provider) that you intend to have a private connection to.
Once the VCN is created, you can change, add, and remove its CIDR blocks.
When designing subnets, consider where your traffic is going and your security needs. Connect all resources within a specific layer or role to the same subnet, which can serve as a security boundary.
Cloud GuardClone and customize the default recipes provided by Oracle to create custom detector and responder recipes. These recipes enable you to specify what type of security breach generates a warning and what actions are allowed to be taken on it. For example, you might want to detect an Object Storage bucket that has visibility set to public.
Apply Cloud Guard at the tenant level to cover the broadest reach and reduce the administrative burden of maintaining multiple configurations.
You can also use the Managed List feature to apply certain configurations to detectors.
OCI functionThe OCI functions in this architecture are developed using Python. Note that OCI functions support a variety of programming languages, and you can choose your preferred language to develop and deploy your functions.
OCI Stream**A stream pool is created in a private subnet of the VCN. Flows are created using this stream pool. Make sure that you create the function in the same VCN as the private OSS stream and have all the necessary access permissions.
Stream** also leverages the Kafka Connect ecosystem to interface directly with first- and third-party products, through the use of off-the-shelf Kafka source and sink connectors. For more information about using Kafka Connect, see Explore More.
OCI Object StorageThis architecture uses standard object storage to upload files. Make sure to enable "Emit Object Events" so that even services can be triggered. For sensitive data, we recommend that you use private buckets. OCI eventsIn this schema, OCI events are configured to listen for Object Creation and Object Update events in the OCI Object Store. The service is called after the object is uploaded to the OCI object store, and a function is called to process it. Considerations:When implementing this reference architecture, it is important to consider the following aspects. PerformanceOCI events, OCI functions, and OCI streams** are highly scalable. Depending on your expected file size and number, consider adjusting the number of partitions and streams. SecurityUse policies to restrict who can access OCI resources. For OCI object storage, encryption is enabled by default and cannot be disabled. Access to all functions deployed in OCI functions is controlled through Oracle Cloud Infrastructure Identity and Access Management (IAM), which allows function management and function invocation permissions to be assigned to specific users and user groups. It is recommended to store secret and sensitive data in an OCI vault. Consider using an OCI vault to store API keys and authentication tokens used to authorize with OCI services. AvailabilityOracle ensures high availability of OCI functions, OCI events, OCI streams**, and OCI logs, all cloud-native and fully managed. OCI Streaming** includes the following high availability capabilities: Continuous log data streaming.
A multi-threaded and horizontally scalable service.
Ingestion in near real-time.
Resilience to short-term outages.
Optimized for efficient data usage.
CostYou only pay for the resources that your function uses when it runs. DeploymentThe GitHub repository contains OCI functions that implement this reference architecture. Visit GitHub.
Clone or repositorate to your local computer.
Follow the guidance in the README documentation.
As an Oracle Premier Partner, Agilewing is redefining the way enterprises experience Oracle Cloud Services. With its streamlined account opening process and best-in-class technical support, Agilewing transforms the complex process of account opening and operation into an easy, intuitive experience. With our one-stop shop, you can quickly get up and running with the full range of Oracle Cloud services, so you can seamlessly integrate into the cloud. Agilewing's AgileCDN service, combined with OCI's cloud-based services, provides a best-in-class global content acceleration solution. A strong network of more than 2,800 global POP nodes and 7,000 direct connection points ensures efficient and stable operation no matter where your business expands to the world. Leveraging the advanced technology of Oracle Cloud, Agilewing is committed to simplifying the process of cloud service building, cloud migration, and business going global. "Our partnership model provides customers with cost-effective solutions that allow them to focus more on their core business while enjoying the high performance and security of Oracle Cloud." Oracle Cloud Service, as a promising field, opens the door to new opportunities for enterprises with its high performance, security, and globally consistent service standards. Through Agilewing's professional services, both individual users and enterprises can easily enter this new era full of technological innovation and high performance. Let Agilewing start exploring Oracle Cloud Services and open the door to a whole new world today.