Package edu.ie3.datamodel.io.source.csv

Class CsvDataSource

java.lang.Object
edu.ie3.datamodel.io.source.csv.CsvDataSource
Direct Known Subclasses:
CsvGraphicSource, CsvIdCoordinateSource, CsvRawGridSource, CsvResultEntitySource, CsvSystemParticipantSource, CsvThermalSource, CsvTimeSeriesMappingSource, CsvTimeSeriesMetaInformationSource, CsvTimeSeriesSource, CsvTypeSource, CsvWeatherSource
public abstract class CsvDataSource extends Object
Parent class of all .csv file related sources containing methods and fields consumed by allmost all implementations of .csv file related sources.
Since:
05.04.20

  • Field Details

  • Constructor Details

  • Method Details

    • parseCsvRow

      protected String[] parseCsvRow(String csvRow, String csvSep)
      Parse a given row of a valid RFC 4180 formatted csv row
      Parameters:
      csvRow - the valid row
      csvSep - separator of the csv file
      Returns:
      an array with the csv field values as strings

    • isPresentCollectIfNot

      protected <T extends UniqueEntity> Predicate<Optional<T>> isPresentCollectIfNot(Class<? extends UniqueEntity> entityClass, ConcurrentHashMap<Class<? extends UniqueEntity>,LongAdder> invalidElementsCounterMap)
      Returns a predicate that can be used to filter optionals of UniqueEntitys and keep track on the number of elements that have been empty optionals. This filter let only pass optionals that are non-empty. Example usage: 
      
 Collection.stream().filter(isPresentCollectIfNot(NodeInput.class, new ConcurrentHashMap<>()))
      Type Parameters:
      T - the type of the entity
      Parameters:
      entityClass - entity class that should be used as they key in the provided counter map
      invalidElementsCounterMap - a map that counts the number of empty optionals and maps it to the provided entity clas
      Returns:
      a predicate that can be used to filter and count empty optionals

    • printInvalidElementInformation

      protected void printInvalidElementInformation(Class<? extends UniqueEntity> entityClass, LongAdder noOfInvalidElements)

    • saveMapGet

      protected String saveMapGet(Map<String,String> map, String key, String mapName)

    • logSkippingWarning

      protected void logSkippingWarning(String entityDesc, String entityUuid, String entityId, String missingElementsString)

    • findFirstEntityByUuid

      protected <T extends UniqueEntity> Optional<T> findFirstEntityByUuid(String entityUuid, Collection<T> entities)
      Returns an Optional of the first UniqueEntity element of this collection matching the provided UUID or an empty Optional if no matching entity can be found.
      Type Parameters:
      T - type of the entity that will be returned, derived from the provided collection
      Parameters:
      entityUuid - uuid of the entity that should be looked for
      entities - collection of entities that should be
      Returns:
      either an optional containing the first entity that has the provided uuid or an empty optional if no matching entity with the provided uuid can be found

    • buildStreamWithFieldsToAttributesMap

      protected Stream<Map<String,String>> buildStreamWithFieldsToAttributesMap(Class<? extends UniqueEntity> entityClass, CsvFileConnector connector)
      Tries to open a file reader from the connector based on the provided entity class and hands it over for further processing.
      Parameters:
      entityClass - the entity class that should be build and that is used to get the corresponding reader
      connector - the connector that should be used to get the reader from
      Returns:
      a parallel stream of maps, where each map represents one row of the csv file with the mapping (fieldName to fieldValue)

    • buildStreamWithFieldsToAttributesMap

      protected Stream<Map<String,String>> buildStreamWithFieldsToAttributesMap(Class<? extends UniqueEntity> entityClass, BufferedReader bufferedReader)
      Reads the first line (considered to be the headline with headline fields) and returns a stream of (fieldName to fieldValue) mapping where each map represents one row of the .csv file. Since the returning stream is a parallel stream, the order of the elements cannot be guaranteed.
      Parameters:
      entityClass - the entity class that should be build
      bufferedReader - the reader to use
      Returns:
      a parallel stream of maps, where each map represents one row of the csv file with the mapping (fieldName to fieldValue)

    • csvRowFieldValueMapping

      protected List<Map<String,String>> csvRowFieldValueMapping(BufferedReader reader, String[] headline)

    • distinctRowsWithLog

      protected Set<Map<String,String>> distinctRowsWithLog(Collection<Map<String,String>> allRows, Function<Map<String,String>,String> keyExtractor, String entityDescriptor, String keyDescriptor)
      Returns a collection of maps each representing a row in csv file that can be used to built one entity. The uniqueness of each row is doubled checked by a) that no duplicated rows are returned that are full (1:1) matches and b) that no rows are returned that have the same composite key, which gets extracted by the provided extractor. As both cases destroy uniqueness constraints, an empty set is returned to indicate that these data cannot be processed safely and the error is logged. For case a), only the duplicates are filtered out and a set with unique rows is returned.
      Parameters:
      allRows - collection of rows of a csv file an entity should be built from
      keyExtractor - Function, that extracts the key from field to value mapping, that is meant to be unique
      entityDescriptor - Colloquial descriptor of the entity, the data is foreseen for (for debug String)
      keyDescriptor - Colloquial descriptor of the key, that is meant to be unique (for debug String)
      Returns:
      either a set containing only unique rows or an empty set if at least two rows with the same UUID but different field values exist

    • getAssetType

      protected <T extends AssetTypeInput> Optional<T> getAssetType(Collection<T> types, Map<String,String> fieldsToAttributes, String skippedClassString)
      Checks if the requested type of an asset can be found in the provided collection of types based on the provided fields to values mapping. The provided fields to values mapping needs to have one and only one field with key TYPE and a corresponding UUID value. If the type can be found in the provided collection based on the UUID it is returned wrapped in an optional. Otherwise an empty optional is returned and a warning is logged.
      Type Parameters:
      T - the type of the resulting type instance
      Parameters:
      types - a collection of types that should be used for searching
      fieldsToAttributes - the field name to value mapping incl. the key TYPE
      skippedClassString - debug string of the class that will be skipping
      Returns:
      either an optional containing the type or an empty optional if the type cannot be found

    • assetInputEntityDataStream

      protected <T extends AssetInput> Stream<AssetInputEntityData> assetInputEntityDataStream(Class<T> entityClass, Collection<OperatorInput> operators)
      Returns a stream of optional AssetInputEntityData that can be used to build instances of several subtypes of UniqueEntity by a corresponding EntityFactory that consumes this data.
      Type Parameters:
      T - type of the entity that should be build
      Parameters:
      entityClass - the entity class that should be build
      operators - a collection of OperatorInput entities that should be used to build the data
      Returns:
      stream of optionals of the entity data or empty optionals of the operator required for the data cannot be found

    • assetInputEntityDataStream

      protected <T extends AssetInput> AssetInputEntityData assetInputEntityDataStream(Class<T> entityClass, Map<String,String> fieldsToAttributes, Collection<OperatorInput> operators)

    • nodeAssetInputEntityDataStream

      protected Stream<Optional<NodeAssetInputEntityData>> nodeAssetInputEntityDataStream(Stream<AssetInputEntityData> assetInputEntityDataStream, Collection<NodeInput> nodes)
      Returns a stream of optional NodeAssetInputEntityData that can be used to build instances of several subtypes of UniqueEntity by a corresponding EntityFactory that consumes this data. param assetInputEntityDataStream
      Parameters:
      assetInputEntityDataStream - a stream consisting of AssetInputEntityData that is enriched with NodeInput data
      nodes - a collection of NodeInput entities that should be used to build the data
      Returns:
      stream of optionals of the entity data or empty optionals of the node required for the data cannot be found

    • nodeAssetEntityStream

      protected <T extends AssetInput> Stream<Optional<T>> nodeAssetEntityStream(Class<T> entityClass, EntityFactory<T,NodeAssetInputEntityData> factory, Collection<NodeInput> nodes, Collection<OperatorInput> operators)
      Returns a stream of optional entities that can be build by using NodeAssetInputEntityData and their corresponding factory.
      Type Parameters:
      T - Type of the AssetInput to expect
      Parameters:
      entityClass - the entity class that should be build
      factory - the factory that should be used for the building process
      nodes - a collection of NodeInput entities that should be used to build the entities
      operators - a collection of OperatorInput entities should be used to build the entities
      Returns:
      stream of optionals of the entities that has been built by the factor or empty optionals if the entity could not have been build

    • simpleEntityDataStream

      protected <T extends ResultEntity> Stream<SimpleEntityData> simpleEntityDataStream(Class<T> entityClass)
      Returns a stream of SimpleEntityData for result entity classes, using a fields-to-attributes map.
      Type Parameters:
      T - Type of the ResultEntity to expect
      Parameters:
      entityClass - the entity class that should be build
      Returns:
      stream of SimpleEntityData