Early automated SITS analysis tools used traditional machine learning techniques [19]. However, deep convolutional networks have since enabled the extraction of more detailed spatial features [20]. Initially, temporal aspects were addressed through handcrafted temporal descriptors or probabilistic models, which have been effectively superseded by architectures such as recurrent, convolutional, or differential networks. More recently, attention-based methods have also been introduced. It has been adapted to encode sequences of remote-sensing images and has led to significant progress in pixel-wise and parcel-wise classification. In parallel, hybrid architectures relying on U-Net-type architectures for encoding the spatial dimension and recurrent networks for the temporal dimension are well suited for the semantic segmentation of SITS. In this paper, we propose to combine this hybrid architecture with the promising temporal attention mechanism.

The initial step in panoptic segmentation is identifying individual instances, known as instance segmentation. Most instance segmentation methods in remote sensing focus on single-image acquisitions [21]. For example, several techniques have been developed to detect individual instances of trees, buildings, or fields. Some approaches begin with a border detection step and require postprocessing to define distinct cases. Other methods use segmentation as a preliminary step to compute cluster-based features, though they do not provide clear mappings of clusters to specific objects. Petitjean and Dacher [22] introduced a segmentation-aided classification method for image time series; however, their approach segments each image separately without consistently identifying objects across the sequence. This paper presents the first end-to-end framework designed to perform semantic and instance segmentation on SITS jointly [23].

In extensive research on instance segmentation, Mask-RCNN is a leading method for natural images. Recently, Wang et al. [24] introduced CenterMask, a lighter and more efficient single-stage approach, which serves as a foundation for our work in this paper. Several methods aim to extend instance or panoptic segmentation techniques from images to video. However, as discussed in the introduction, SITS differ significantly from natural videos, necessitating specialized algorithmic and architectural modifications.

We consider a convolutional en-coder ε with L levels 1, ..., L. Each level is composed of a sequence of convolutions, Rectified Linear Unit (ReLu) activations, and normalizations. Except for the first level, each block starts with a strided convolution, dividing the resolution of the feature maps by a factor 2.

For each time stamp t simultaneously, the encoder εt at level / takes as input the feature map of the previous

t−1 t

level et , and outputs a feature map et, of size C x H x Wi with H1 = H/2 − 1 and W = W/2 − 1. The resulting feature maps are then temporally stacked into a feature map sequence e of size Tx Cix H x W:

t t−1 T

e = [e (e )] t=0 t ∈ [1, L]. (1) With e = X and [0] the concatenation operator along the temporal dimension. When constituting batches, we flatten the temporal and batch dimensions. Since each sequence comprises images acquired at different times, the batches' samples are not identically distributed. To address this is- sue, we use Group Normalization with 4 groups instead of Batch Normalization in the encoder.

I. Satellite imagery: Acquire high-resolution satellite images from sources like Sentinel-1, Sentinel-2, Landsat, or MODIS [25].

II. Weather data: Collect meteorological data, including temperature, precipitation, humidity, and wind speed, from weather stations or APIs [26].

III. Soil data: Obtain soil maps and sampling data to characterize soil properties [27].

IV. Data cleaning: Handle missing values, outliers, and inconsistencies in the data.

V. Data preprocessing: Apply image processing techniques (e.g., atmospheric correction, geometric correction) and feature extraction (e.g., vegetation indices, texture analysis) to prepare the data for modeling.

I. Feature engineering: Create relevant features from the preprocessed data, such as vegetation indices, texture features, and temporal trends.

II. Model selection: Choose appropriate machine learning or deep learning algorithms (e.g., random forest, support vector regression, gradient boosting, convolutional neural networks, recurrent neural networks) based on data characteristics and problem complexity.

III. Model training: Train the selected model(s) on the prepared dataset, optimizing hyperparameters for optimal performance.

IV. Model evaluation: Evaluate the trained model(s) using appropriate metrics (e.g., mean squared error, root mean squared error, R-squared) and cross-validation techniques.

I. Deployment: Deploy the trained model(s) to a suitable platform (e.g., cloud-based or local server) for real-time or near-real-time yield predictions.

II. Visualization: Develop user-friendly interfaces (e.g., web applications, dashboards) to visualize the input data, model predictions, and uncertainty estimates.

I. Data availability: Ensure access to reliable and timely satellite imagery, weather data, and soil information II. Computational resources: Utilize sufficient computational resources (e.g., cloud-based computing) to handle large datasets and complex models III. Algorithm complexity: Select algorithms that balance accuracy and computational efficiency IV. Data quality: Address data quality issues, such as noise, missing values, and outliers

I. Collect satellite imagery, weather data, and soil data from various sources.

II. Clean and preprocess the data to remove noise, inconsistencies, and missing values.

III. Extract relevant features from the data, such as vegetation indices, texture features, and temporal trends.

I. Select and train appropriate machine or deep learning models on the preprocessed data.

II. Evaluate the performance of the trained models using appropriate metrics.

III. Optimize hyperparameters to improve model accuracy and generalization.

I. Deploy the trained model to a suitable platform for real-time or near-real-time predictions.

II. Develop user-friendly interfaces to visualize input data, model predictions, and uncertainty estimates. Following this system design, we aim to develop a robust and efficient agricultural yield prediction system that can provide valuable insights to farmers, policymakers, and researchers [28].

We introduce PASTIS, the first large-scale, publicly accessible SITS dataset featuring semantic and panoptic annotations1.

The PASTIS dataset includes 2,433 sequences of multispectral images, each sized 10 x 128 x 128 pixels. Each sequence contains 38 and 61 observations taken from September 2018 to November 2019, totaling over 2 billion pixels.

Acquisition intervals are irregular, averaging 5 days due to automated cloud cover filtering by the satellite provider THEIA. The 10 channels correspond to the Sentinel-2 satellite's non-atmospheric spectral bands, processed with atmospheric correction and resampled to a 10-meter spatial resolution per pixel. This dataset covers roughly 4,000 square kilometers across four distinct regions of France with different climates and crop types, representing about 1% of the French Metropolitan area. Approximately 28% of the images have some degree of cloud cover.