Generative AI Applied to Digital Twins: Towards Improving Intelligent Systems in Industry

3.1 On-industry experimentation

The concept of OIE is presented as a new manifestation of collaborative experimental research. At its core is a potential global community that makes it possible to build productive relationships between industry professionals, R&D, scientists, and startups to develop the innovation paths needed to solve the challenges of the manufacturing industry.

OIE is the result of accumulated improvements in several domains that collaboratively realize a transformational change in recognizing the power of collective intelligence; that is, industries mainly depend on their internal R&D departments to conceive and develop new products and bring them to market; however, open innovation emphasizes collaboration and partnerships with researchers, science and technology institutions, and startups, leveraging external knowledge and contributions.

In the context of OIE, open innovation adopts a more inclusive and integrated approach to the investigation and experimentation environment, actively seeking collaborative approaches to the use of real and real-time data and the application of solutions in the manufacturing production environment through virtual representations of the real scenario. Frequently, this change is catalyzed by the analytical, learning, and autonomous decision support opportunities presented by digital technologies such as digital twins and artificial intelligence applied in cyber-physical systems.

OIE is defined as an innovation process that brings together interested industry stakeholders around mutually beneficial experimentation to support the industry’s own operation, management, and innovation decisions. This purpose is supported by mechanisms that are based on the complex and intertwined plot of formal and participatory applied research in the industry’s RD&I area.

OIE allows RD&I to occur directly in production processes and at significant scales, rather than in minimized representations of the real environment that are designed externally. The interests of industry and external OIE actors are explicitly recognized as a prerequisite for negotiating their alignment and building productive collaborative relationships. Experimentation in OIE RD&I is understood as a deliberate process of joint exploration through which researchers and others engage closely with the realities of industry to align with their needs, enabled by a virtual environment that faithfully represents the scenario of the production process.

OIE encompasses the heterogeneity of circumstances, practices, and demands of the manufacturing industry, providing real and contextualized information on how to collaborate, predict critical scenarios, and develop local innovations. Researchers and other interested parties add value to the experimental process by providing specialized skills and external perspectives to help local RD&I evaluate ideas in their real scenarios, where the empirical knowledge and experiential learning of industry professionals are complemented by the suggestion of metrics and experimental designs, performing analyses and documenting experiences, interpreting results and expanding horizons, proposing opportunities and next steps in the open innovation process.

The aim is thus to leverage the industry’s own knowledge, harness collective intelligence under the focus of the external perspective of other experts, and create value for all by stimulating the production of new insights through co-learning and the application of solutions through the hybridization of knowledge enabled by intelligent systems based on virtualized environments.

3.2 Digital twins

OIE is enabled by intelligent systems that are based on the idea of digital twin scenarios through which we can operate a system in cloud computing as a virtual representation for management and analysis of industrial databases based on IoT (Internet of Things) produced through sensing and industrial procedures [14, 17].

The use of DT technologies in OIE allows for the visualization and understanding of processes and physical assets in real-time and the implementation of solutions that integrate these mechanisms through an iterative, collaborative, and flexible process that can be applied and validated virtually and subsequently in the real scenario.

The use of DT for the development of intelligent systems in the manufacturing process in OIE can be applied in (i) status monitoring in which digital replicas of physical assets are used in which machines are continuously monitored using IoT and sensing [18]; (ii) simulation of the virtualization of machines, processes, and products that are created to represent real configurations. Simulation allows for the joint proposition of design, development, and testing of new products and processes using their digital representation before applying to real physical assets [19]; and (iii) visualization of physical assets, from dashboards and data analytics with real-time data and alert systems to monitor and debug an operational problem.

In the OIE environment, DTs are considered as an exact replica of physical assets that enable value-added services, built from the collaborative investigation of industry R&D&I teams, researchers, and startups, to apply autonomous intelligent agents to the represented assets, without the need for physical access to the production environment.

In the context of intelligent systems that use cyber-physical scenarios to support the deployment of OIE, DTs offer access and communication, analysis, and intelligence capabilities [18]. The access and communication layer are responsible for interacting with production processes and gaining access to data through sensors and open interfaces on the status of a physical asset to update its representation in the DT. The analysis layer provides analytical capabilities of physical assets and their processes and can perform additional analysis tasks on data collected in real-time to help in the decision-making process, converting raw sensory input into actionable knowledge. The intelligence layer enables the ability to perform complex decision-making using large dataset processing, domain knowledge, and historical knowledge bases. It is responsible for DT’s intervention in the represented physical environment and executes autonomous decision-making in that environment.

In intelligent systems, DTs operate as cyber-physical agents capable of accessing, analyzing, and interpreting the current state of their physical counterpart, enhanced using GAI. Faced with irregularities or opportunities for continuous improvement and innovation, the DT can interact directly with the operational environment, performing virtual simulations and executing processes adaptively. This capability allows for the identification of optimized solutions before their implementation in the real environment, reducing risks, improving production efficiency, and driving innovation in industrial systems.

3.3 Generative artificial intelligence integrated with digital twins

Generative artificial intelligence as a computational technology that can generate content, solutions, and insights from a large dataset, which goes beyond simply analyzing or acting on existing data in intelligent systems, with the aim of assisting or replacing humans in creating diverse, personalized, and scalable content more quickly and at a lower cost [20], has the potential to optimize and maximize the results of intelligent systems based on digital twins in the context of OIE.

In the industrial sector, the main advances in GAI are in the use of models trained with large sets of historical and real-time private data generated from cyber-physical sensing systems.

Similarly, based on datasets generated by DT systems from production environments, real-time perception and acquisition of static data, produced by a network of sensors in IoT environments, advanced GAI technologies enable the generation of diverse content and solutions in predicting failures, mitigating operational problems, and proposing improvements in the production process while intensifying the validation of collaborative experimentation from dialogs in open innovation processes, the basis for the adoption of the OIE concept.

The application of GAI in DT in the industrial sector in OIE scenarios represents a fundamental evolution for the creation of intelligent systems that extend reflections of their physical representations with predictive and adaptive functionalities [21]. Advanced connectivity and pre-training techniques for personalized models highlight a coordinated evolution towards a deeper and more meaningful integration. This integration ensures that digital twins can leverage the full spectrum of generative AI potential, from improving real-time decision-making to facilitating long-term strategic planning [22].

The integration of GAI in DT marks a fundamental advance in the construction of intelligent systems that enable the implementation of on-industry experimentation processes. In an open manufacturing innovation environment, intelligent systems can be improved by allowing users to interact in natural language, generating high-quality knowledge in the context of planning or operating manufacturing processes [10]. DT technology offers validation and reliability support for the generated content, mitigating the challenge of low reliability in GAI models. This establishes the technological basis for RD&I professionals and researchers to collaboratively advance in intelligent systems based on the integration of digital twins driven by GAI.

4.1 Characteristics of the IXIS framework

The processes of continuous improvement, problem-solving, and innovation in manufacturing processes are characterized by investigation and experimentation in cyber-physical environments that integrate physical assets and digital systems that act in the integration of production lines, management and remote control, modularity, real-time operation, virtualization, and service orientation. The conceptual framework presented is based on the proposal of intelligent systems that use the digital representation of the instance of a physical asset (machine, production line, product, etc.), allowing, through computer-assisted content generation, collaboration and collective construction of intelligent solutions for the industrial system to occur.

From the integration of DT and GAI in intelligent systems, IXIS presents the following characteristics:

1. Collaboration: from the perspective that a virtual representation of a production system is made available, industry professionals and external stakeholders can contribute to the proposal of solutions in an integrated way.

2. Virtualization: using DT, improvements and innovations can be validated in a digital environment before being applied in the real environment.

3. Data-oriented: with the integration between the real and virtual environment happening in real-time, a large volume of data is made available to be processed and used for insight generation.

4. Decentralization: the virtualization of physical assets allows access and intervention by various stakeholders in production environments in person or remotely.

5. Autonomy: from GAI algorithms applied in large datasets, intelligent processes can act autonomously in production systems with the virtualized validation of the DT.

4.2 Definition of the IXIS framework

The IXIS conceptual framework is a cyber-physical system that, through an intelligent system that integrates DT technologies and GAI models, creates a collaborative experimentation environment in industrial production processes.

From access to the status of the production environment in real-time and the generation of content and insights based on large datasets, together with virtualization, simulation, and real-time verification, the intelligent system allows for integration between actors in the industrial innovation process. This integration enables investigation and experimentation enabled by data management, human-computer interaction, process knowledge from a large volume of historical and real-time data, verification and optimization of feedback, and generation of insights and contributions to innovation. Its main objective is to provide an open innovation model in a real environment and optimize the productivity and quality improvement of production processes.

From this definition, the IXIS architecture is presented as shown in Figure 1, comprising functional modules, namely, (i) physical layer; (ii) control layer; (iii) digital twin layer; (iv) generative artificial intelligence layer; and (v) user layer.

4.3 Functional modules of the IXIS framework

The physical layer is an essential module in a cyber-physical manufacturing system, representing the physical assets of the production environment as tangible elements that can be monitored, controlled, or influenced through information and actions in the digital world. These assets may include machines, equipment, production processes, and products, among others.

In the context of an intelligent system enabled by DT, the physical layer is equipped with IoT devices that sense and continuously monitor the state and performance of these assets. These sensors collect real-time data and signals, such as temperature, vibration, pressure, and other variables critical to the production process.

The data generated by IoT devices is transmitted to an intelligent system, where it will be digitally represented for (i) DT construction and (ii) composing a large dataset for GAI technologies.

The control layer acts as a logical module in the intelligent system, performing the monitoring, modeling, and processing of data and signals obtained in real-time from the physical layer, complements the data acquisition of physical assets through computer vision systems that use cameras, sensors, and image processing algorithms to visualize and interpret the real world, allowing to expand the synchronized status of the physical asset through the informational representation of the asset based on visual information.

The control layer also acts in the intervention of the physical asset from instructions and commands generated by the integration of DT and GAI that allow for prediction, simulation, optimization, and evaluation in the execution of commands in the physical layer.

The DT layer is defined as a complex and interconnected module that emerges as a crucial element in the architecture of the proposed intelligent system. Integrated with the generative artificial intelligence layer, it articulates between the physical environment and the digital environment, creating the interface between system expert users and the physical production environment.

This layer is the virtual and dynamic representation of the physical layer, being fed by data continuously collected from the physical asset through sensors, control systems, and other sources of information.

The synthetic representation of the physical world can be made by three-dimensional models enabling immersive visualization that expands the understanding of the real environment. Using virtual reality and augmented reality technologies, users visualize and interact with physical assets intuitively and immersively, as if they were physically present on-site.

In addition to the sensed data of the physical asset, this layer generates synthetic data to simulate scenarios, test hypotheses, and predict the behavior of the physical asset in different operating conditions from generative data of the GAI layer to automatically generate improvement recommendations, optimize processes, diagnose problems, and even design new products or processes, presented through intelligent dashboards that provide insights into asset performance, identify bottlenecks, alert to possible failures and anomalies, assist in decision-making, support predictive maintenance and production optimization, and allow for simulating the impact of these insights in the virtual environment before being applied to the physical layer.

The GAI layer acts as an engine of creation and optimization, driving strategic decision-making and continuous experimentation. This process happens because GAI is capable of processing large volumes of data, both historical and real-time, coming from DT and the physical layer. This capability allows for identifying patterns, trends, and anomalies that would be difficult to detect by other means. The generative AI layer uses pre-training and fine-tuning techniques of machine learning models to continuously adapt to changes in the industrial environment and improve its ability to generate relevant and accurate content.

Various GAI models can be applied to enhance different aspects of the intelligent system. Here are some specific examples:

• Small language models (SLM): These can be used to generate personalized outputs within the corporate domain, ensuring that sensitive information is protected. For instance, in a manufacturing setting, SLM can generate specific reports or analyses related to production processes, maintaining data privacy. SLM are language models trained in specific types of data to produce personalized outputs that keep data within the corporate domain, protecting sensitive information, the use of SLM instead of LLM is suggested in this layer, because LLM can also generate incorrect answers (hallucinations) due to the breadth of information they process. As the proposed intelligent system operates in a specific domain such as a manufacturing industry, SLM are more adherent to the characteristics and demands of this environment.

• Large language models (LLM): While the framework suggests using SLM for data privacy, LLMs can still play a role in tasks that require a broader understanding of language and context. For example, they can assist in generating comprehensive documentation for new products or processes, or in providing detailed analyses of industry trends and how they might impact the manufacturing process.

• Generative adversarial networks (GANs): GANs can be employed to create synthetic data that mimics real-world data from the production process. This synthetic data can be valuable for training other machine learning models or for simulating rare events or failure scenarios in the production line, helping to improve the system’s resilience and predictive capabilities.

• Diffusion models: These models can be used to generate high-quality images or 3D models of products or production line layouts. This can aid in the design and visualization of new products or in optimizing the physical layout of the manufacturing facility.

The GAI layer can be integrated with intelligent agents that act as interfaces between the digital and physical worlds. These agents can receive commands from the GAI, execute tasks in the physical layer, and provide feedback to the DT, creating a cycle of learning and continuous improvement. This layer can generate creative and relevant content for various purposes, such as 3D models of products, process simulations, analysis reports, improvement recommendations, and even new solutions to industrial problems. It can also generate virtual models of industrial processes based on historical and real-time data, allowing for simulating different scenarios and optimizing process performance even before they are implemented in the physical world.

In the context of innovation, the GAI layer can generate new solutions to industrial problems based on preexisting data and business rules, driving innovation and continuous improvement of processes.

The GAI layer, when applied to an intelligent system composed of a DT in the context of a cyber-physical environment in the manufacturing industry, represents a significant advance in the search for more efficient, innovative, and adaptable processes. By enabling modeling, simulation, evaluation, and innovation of industrial processes, GAI empowers companies to make more strategic decisions and reach new levels of manufacturing excellence.

In an advanced industrial ecosystem with the characteristics of collaborative experimentation for open innovation, the user layer serves as the crucial interface that allows human interaction with a complex intelligent system, composed of DT, GAI in a cyber-physical environment. This layer, composed of startups, APIs, RD&I professionals, and industry researchers, plays a fundamental role in the validation, homologation, and collaborative implementation of strategic decisions.

The user layer offers intuitive and accessible interfaces, allowing various stakeholders to interact with the intelligent system. They can simulate complex scenarios in the virtualized environment of the DT, testing different hypotheses and validating the effectiveness of new strategies. The user layer allows for the homologation of decision-making in a safe and controlled environment, before being applied in the physical layer.

Based on the insights and results obtained, users can generate commands for the GAI layer and for the DT layer. These commands may include requests to generate new models, simulate processes, optimize parameters, or even create new products.

The user layer acts as an innovation laboratory, where startups, researchers, and industry professionals can collaborate to develop new solutions and improve existing processes. User feedback is fundamental for the continuous improvement of the intelligent system, ensuring that it meets the needs of the industry.

5.1 Functional modules

The IXIS framework is structured into five functional modules:

1. Physical layer: This layer forms the foundation, comprising the physical assets within the production environment, such as machines, equipment, and IoT devices. These IoT devices are crucial for collecting real-time data on various parameters like temperature, pressure, vibration, and so on, providing a continuous stream of information about the operational status of physical assets.

2. Control layer: This layer acts as an intermediary, responsible for monitoring and processing the data received from the Physical Layer. It also plays a key role in controlling physical assets based on insights and commands generated by the DT and GAI layers. The Control Layer may also incorporate computer vision systems to enhance data acquisition and monitoring.

3. Digital twin (DT) layer: This layer creates a virtual representation of the physical assets and processes. It continuously receives data from the Physical and Control Layers to maintain an accurate and up-to-date reflection of the real-world environment. The DT layer is crucial for simulation, scenario testing, and generating insights through dashboards and data analytics.

4. Generative AI (GAI) layer: This is where the power of AI is harnessed. The GAI layer processes large volumes of data to identify patterns, generate predictions, and create various forms of content, such as 3D models, process simulations, and improvement recommendations. It employs machine learning models, and the framework suggests the use of Small Language Models (SLM) for enhanced data privacy.

5. User layer: This layer provides an interface for human users to interact with the IXIS framework. It enables stakeholders, including R&D professionals, industry researchers, and startups, to access data, visualize simulations, validate strategies, and send commands to the GAI and DT layers.

5.2 Use case scenarios

Based on the description of the IXIS framework, here are some illustrative use case scenarios in specific industrial settings:

1. Optimizing a production line:

   • Scenario: A manufacturing plant wants to optimize its assembly line for increased efficiency and reduced bottlenecks.

   • IXIS framework application:

     • Physical layer: IoT sensors on each machine collect real-time data on cycle times, operational status, and potential delays. Computer vision in the Control Layer monitors the flow of materials and identifies any anomalies or stoppages.

     • Digital twin layer: A virtual model of the assembly line is created and continuously updated with the data from the Physical and Control Layers.

     • Generative AI layer: Using historical and real-time data, GAI models (potentially GANs or simulation-based models) can generate various scenarios with different configurations of the production line, such as adjusting the sequence of tasks, the number of workstations, or the placement of equipment. SLMs could provide concise reports on the predicted outcomes of these scenarios.

     • User layer: Engineers can interact with Digital Twin, visualize the GAI-generated optimization scenarios, and simulate their impact on key performance indicators (KPIs) like throughput and resource utilization. They can then select the optimal configuration and send commands through the Control Layer to implement the changes in the physical production line.

2. Predicting equipment failure (predictive maintenance):

   • Scenario: A large-scale industrial facility wants to minimize downtime caused by unexpected equipment failures.

   • IXIS framework application:

     • Physical layer: Sensors embedded in critical machinery continuously monitor parameters like temperature, vibration, pressure, and electrical current.

     • Control layer: This layer aggregates and preprocesses the sensor data, potentially using edge computing for immediate anomaly detection.

     • Digital twin layer: A digital replica of each critical piece of equipment is maintained, reflecting its current operational status and historical performance data.

     • Generative AI layer: By analyzing historical failure data and the real-time sensor readings, GAI models (potentially time-series forecasting models or anomaly detection algorithms) can predict potential equipment failures and estimate the remaining useful life. GANs could generate synthetic failure data to improve the training of these predictive models, especially for rare failure modes. SLMs can generate alerts and reports summarizing the predicted failures and recommended maintenance actions.

     • User layer: Maintenance personnel receive timely alerts about potential equipment failures through the User Layer interface. They can access Digital Twin to visualize the equipment’s condition, review the GAI-generated predictions and recommendations, and schedule proactive maintenance, reducing unplanned downtime and maintenance costs.

These examples illustrate how the IXIS framework, with its integrated layers and the application of different GAI models, can provide valuable insights and enable proactive decision-making in industrial settings, leading to improved efficiency, reduced costs, and enhanced operational resilience. The concept of OIE is also supported by this framework, allowing for the safe exploration and validation of new strategies within the digital twin environment before physical implementation.

5.3 Challenges and limitations

While the IXIS framework offers significant potential, its implementation also presents several challenges and limitations:

• Data requirements and management: The effectiveness of the IXIS framework heavily relies on the availability of high-quality, real-time data. Collecting, storing, and processing the massive volumes of data generated by IoT devices and other sources can be a significant challenge. Issues related to data quality, data security, and data integration need to be addressed.

• GAI model reliability and explainability: GAI models, particularly Large Language Models (LLMs), can sometimes produce inaccurate or unreliable results (hallucinations). Ensuring the reliability, accuracy, and explainability of GAI-generated insights is crucial for building trust and enabling effective decision-making. The framework suggests using SLMs to mitigate some of these issues, but further research and validation are needed.

• Computational complexity and scalability: Implementing and running complex DT simulations and GAI models can be computationally intensive, requiring significant processing power and infrastructure. Scaling the IXIS framework to handle large-scale industrial operations with numerous assets and processes can be a major challenge.

• Integration and interoperability: Integrating various systems, devices, and software components from different vendors can be complex. Ensuring interoperability and seamless data exchange between the Physical Layer, Control Layer, DT Layer, GAI Layer, and User Layer is essential for the effective functioning of the IXIS framework.

• Human factors and organizational change: Implementing the IXIS framework requires significant changes in organizational processes, workflows, and roles. Resistance to change from employees and the need for training and upskilling can be challenges. Effective change management strategies are crucial for successful implementation.

• Security and privacy: Protecting sensitive data and ensuring the security of the IXIS framework from cyber threats is paramount. Robust security measures and data privacy protocols must be implemented to safeguard the system and its data.

The IXIS framework provides a promising blueprint for leveraging GAI and DT to drive innovation and optimization in manufacturing through OIE. By enabling collaboration, virtualization, and data-driven decision-making, IXIS has the potential to transform industrial processes and enhance competitiveness. However, addressing the challenges and limitations related to data management, GAI reliability, computational complexity, integration, human factors, and security are crucial for the successful implementation and widespread adoption of this framework.