Technical Manual: Distributed Calculator System with Ice and Kafka - ferpotenUCLM/DistributedSystems GitHub Wiki

1. Introduction

The distributed calculator system is a robust solution that combines the Ice (Internet Communications Engine) middleware with the Apache Kafka streaming platform to provide basic mathematical operations in a decoupled and scalable architecture.

Key Features:

• Basic mathematical operations (addition, subtraction, multiplication, division)
• Asynchronous communication using Kafka
• Error handling (including division by zero)
• Modular and extensible architecture
• Easy deployment and configuration

2. System Architecture

🔧 Used Technologies

• Apache Kafka – Messaging broker for sending/receiving requests and responses.
• ZeroC Ice – RPC middleware for performing remote operations.
• Python / Java / C++ – (Specify what you're using)
• Docker (optional) – For local deployment and testing.

---

🔄 Sequence Diagram in Mermaid

sequenceDiagram
    participant Cliente
    participant Kafka_Req as Kafka (Requests)
    participant Gateway
    participant Ice_Service
    participant Kafka_Res as Kafka (Responses)
    
    Cliente->>Kafka_Req: JSON Request (op1)
    Kafka_Req->>Gateway: Consume mensaje
    Gateway->>Ice_Service: sum(5.0, 10.4)
    Ice_Service-->>Gateway: 15.4
    Gateway-->>Kafka_Res: JSON Response (op1)
    Kafka_Res-->>Cliente: Entrega resultado

Communication Flow

1. Client sends JSON request to topic calculator_requests
2. Gateway consumes messages and executes operation on Ice server
3. Server processes the mathematical operation
4. Gateway sends result to topic calculator_responses
5. Client receives response in JSON format

3. System Requirements

3.1. Software Dependencies

• Python 3.7+
• Apache Kafka 2.8+
• ZeroC Ice 3.7+
• Python Libraries:
• confluent-kafka
• zeroc-ice

3.2. Kafka Configuration

# Create required topics
.\bin\windows\kafka-topics.bat --create --topic calculator_requests --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
.\bin\windows\kafka-topics.bat --create --topic calculator_responses --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

4. System Components

4.1. Ice Server (server.py)

Responsibility: Implements the mathematical calculation logic.

class CalculatorI(Demo.Calculator):
  def sum(self, op1, op2, current=None):
    return op1 + op2

def div(self, op1, op2, current=None):
  if op2 == 0.0:
    raise Demo.DivisionByZero(reason="Division by zero")
    return op1 / op2

Start server:

python server.py

4.2. Kafka Gateway (gateway.py)

Responsibility: Acts as a bridge between Kafka and the Ice server.

class KafkaCalculatorGateway: 
def process_request(self, msg): 
try: 
data = json.loads(msg.value().decode('utf-8')) 
#...validation and processing... 

if operation == "sum": 
result = self.calculator.sum(op1, op2) 
#...other operations... 

response = {"id": req_id, "status": True, "result": result} 
except Exception as e: 
response = {"id": req_id, "status": False, "error": str(e)}

Start gateway:

python gateway.py

4.3. Client (Producer/Consumer)

Request Format:

{
"id": "operation-123",
"operation": "sum",
"args": {
"op1": 5.0,
"op2": 10.4
}
}

Successful Response Format:

{
"id": "operation-123",
"status": true,
"result": 15.4
}

Error Response Format:

{
"id": "operation-456",
"status": false,
"error": "Demo.DivisionByZero: reason: 'Division by zero'"
}

5. Supported Operations

Operation	Name in JSON	Required Parameters
Sum	sum	op1, op2
Subtraction	sub	op1, op2
Multiplication	mult	op1, op2
Division	div	op1, op2

6. Advanced Configuration

6.1. Ice Server Endpoint

Modify in gateway.py:

ice_proxy="calculator:default -p 10000"

6.2. Kafka Configuration

Adjust in gateway.py:

config = {
'bootstrap.servers': 'localhost:9092',
'group.id': 'calculator-group',
'auto.offset.reset': 'earliest'
}

6.3. Port Configuration

Change Ice port in server.py:

adapter = self.communicator().createObjectAdapterWithEndpoints(
"CalculatorAdapter", "default -p 10000")

7. Usage Examples

7.1. Sending Requests

# Sum
echo {"id":"op1","operation":"sum","args":{"op1":5.0,"op2":10.4}} | .\bin\windows\kafka-console-producer.bat --topic calculator_requests --bootstrap-server localhost:9092

# Division (will generate error)
echo {"id":"op2","operation":"div","args":{"op1":10.0,"op2":0.0}} | .\bin\windows\kafka-console-producer.bat --topic calculator_requests --bootstrap-server localhost:9092

7.2. Response Monitoring

.\bin\windows\kafka-console-consumer.bat --topic calculator_responses --bootstrap-server localhost:9092 --from-beginning

Expected Output:

{"id":"op1","status":true,"result":15.4}
{"id":"op2","status":false,"error":"Demo.DivisionByZero: reason: 'Division by zero'"}

8. Error Handling

Common Error Codes

Error	Probable Cause	Solution
DivisionByZero	Attempt to divide by zero	Validate operands before sending
OperationNotSupported	Operation not implemented	Check operation name
MissingOperands	Missing operands in the request	Check JSON structure
ConnectionRefused	Ice server unavailable	Check if server.py is running
KafkaTimeoutError	Kafka is not responding	Check Kafka broker status

9. Testing and Validation

Automated Test Suite

# Testing for valid operations
python -m unittest test_operations.py

# Error handling testing
python -m unittest test_error_handling.py

Manual Testing

# Load testing (run in a separate terminal)
python stress_test.py --requests 1000 --threads 4

10. Future Enhancements

1. Support for advanced operations:

• Boosting
• Roots
• Trigonometric functions

2. Authentication and Security:

• SSL/TLS for communication with Kafka
• Authentication using JWT tokens

3. Monitoring and Logging:

• Prometheus/Grafana integration
• Structured logs in JSON format

4. Horizontal Scalability:

• Load balancing between multiple gateway instances
• Topic partitioning in Kafka

11. Troubleshooting

To start the Kafka broker on your computer so that the gateway can function properly, follow these steps:

---

1. Start Kafka services

Kafka requires Zookeeper (its coordination service) to be running before starting the broker.

A. Start Zookeeper

Open a terminal and run:

# Located in the Kafka installation directory.
bin/zookeeper-server-start.sh config/zookeeper.properties

Note: On Windows, use .bat instead of .sh:

bin\windows\zookeeper-server-start.bat config\zookeeper.properties

B. Start the Kafka broker

In another terminal (with Zookeeper already running), run:

bin/kafka-server-start.sh config/server.properties

(On Windows: bin\windows\kafka-server-start.bat config\server.properties).

---

2. Create the necessary topics

The portal expects two topics:

• calculator_requests (to receive requests).
• calculator_responses (to send responses).

Run in a new terminal:

# Create requests topic
bin/kafka-topics.sh --create --topic calculator_requests --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

# Create response topic
bin/kafka-topics.sh --create --topic calculator_responses --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

(On Windows: bin\windows\kafka-topics.bat).

---

3. Verify that the topics exist

List the topics to confirm:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

The console shows:

calculator_requests
calculator_responses

---

4. Test the message flow

A. Consume responses (for debugging)

In a terminal, subscribe to the response topic:

bin/kafka-console-consumer.sh --topic calculator_responses --bootstrap-server localhost:9092 --from-beginning

B. Send a test request

In another terminal, send a JSON message to the request topic:

echo '{"id":"op1","operation":"sum","args":{"op1":5.0,"op2":10.4}}' | bin/kafka-console-producer.sh --topic calculator_requests --bootstrap-server localhost:9092

If everything works, you'll see the response in the consumer:

{"id": "op1", "status": true, "result": 15.4}

---

Key Notes

1. Startup Order:

• First ZooKeeper, then Kafka.
• If you restart, shut down Kafka first, then ZooKeeper.

2. Common Errors:

• If Kafka doesn't start, verify that Zookeeper is running (netstat -tulnp | grep 2181 on Linux).
• If messages aren't arriving, check that the topics are created correctly.

3. Retention Services:

• For Kafka: Ctrl + C in the broker terminal.
• For Zookeeper: Ctrl + C in your terminal.

Problem: No responses received

Diagnostic steps:

1. Verify that the Ice server is running
2. Verify that the gateway is processing messages
3. Confirm that the topics exist in Kafka:

.\bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

4. Inspect messages in the topics:

.\bin\windows\kafka-console-consumer.bat --topic calculator_requests --bootstrap-server localhost:9092 --from-beginning

Problem: Slow operations

Possible Solutions:

• Increase the number of partitions in Kafka:

.\bin\windows\kafka-topics.bat --alter --topic calculator_requests --partitions 3 --bootstrap-server localhost:9092

• Run multiple gateway instances
• Optimize Ice configuration (thread pool)

12. Conclusion

This system demonstrates a modern architecture that combines Ice's strengths for efficient RPCs with Kafka's scalability and resilience. The decoupled design allows for:

• Independent component upgrades
• Horizontal scalability
• Fault tolerance
• Easy integration with new clients

The system is ready to be deployed in production environments with appropriate security and monitoring configurations.