Commit Graph

604 Commits

Author SHA1 Message Date
Bernhard Kirchen
6da90de765 remove extraction of basic auth params from URL
the extractUrlComponents method did extract username and password
from the URL and encoded it for basic authentication. however, the
respective result string was never used. we only perform basic
authentication if the auth type is "basic" and if username and
password were supplied through the respective inputs.
2024-06-26 21:25:08 +02:00
Bernhard Kirchen
297b149f84 powermeter refactor: generalize HTTP request config
the parameters to peform an HTTP request by the HTTP(S)+JSON power meter
have been generalized by introducing a new config struct. this is now
used for all values which the HTTP(S)+JSON power meter can retrieve, and
also used by the HTTP+SML power meter implementation. we anticipate that
other feature will use this config as well.

generalizing also allows to share serialization and deserialization
methods in the configuration handler and the web API handler, leading to
de-duplication of code and reduced flash memory usage.

a new web UI component is implemented to manage a set of HTTP request
settings.
2024-06-26 21:25:08 +02:00
Bernhard Kirchen
ccba7d8036 move JSON path resolver to Utils class for re-use 2024-06-26 21:25:05 +02:00
Bernhard Kirchen
673b9f4fa8 powermeter refactor: use destructors to de-initialize 2024-06-26 20:51:56 +02:00
Bernhard Kirchen
e78f5849c1 Feature: decode more OBIS values in SML power meters
supersedes #951.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
75c07c17f2 powermeter refactor: SML lib: replace double by float
avoid additional conversions and avoid double for the fact that
calculations on type double are implemented in software, whereas
float is handled in hardware on ESP32.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
6108d24795 powermeter refactor: introduce PowerMeterSml
this new class handles SML data. it uses the SML lib to decode values
and manages those. this de-duplicates code as the class is applicable
to all power meters that collect SML data.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
6e44a6d750 powermeter refactor: allow destruction of httpClient
make sure the wifiClient used by the httpClient lives longer than the
httpClient, as it accesses the pointer to the wifiClient in its
destructor.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
54c04aed61 SDM power meter: remove baud rate setting
this setting was not used. the baud rate for the SDM is set to 9600 in
the source code. until the baud rate being customizable is actually
required by somebody, we remove the setting altogether.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
d99cfd5b31 powermeter refactor: publish values to MQTT in base class
"powertotal" is always published and it is published by the base class
directly. other values are still published by the derived classes, but
use a base class method, which takes care that a common base topic is
used in particular.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
9eb4f1714c powermeter refactor: make timestamp of last update atomic
the timestamp is potentially updated from a different thread, e.g., MQTT
task, than the main loop, which typically reads that timestamp.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
d4c07836d9 MQTT powermeter: avoid iterating subscriptions
instead of iterating a map with subscriptions, we now bind the target
variable to the callback, which is executed once a message is arrived.
this way, the target variable is already linked to the respective topic
when the callback is executed.

lock the mutex when writing the variable, as the MQTT callback is
executed in a different context (MQTT task) than the main loop task,
which otherwise accesses the variables.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
33683d26c8 powermeter refactor: rename providers in enum
the enum values did not change, but their name (only relevant in the
code) are now more expressive.
2024-06-26 20:51:56 +02:00
Bernhard Kirchen
2397e5cdf5 powermeter refactor: split providers into their own classes
it is important to separate the capabilities of each power meter
provider into their own class/source file, as the providers work
fundamentally different and their implementations must not be
intermangled, which made maintenance and improvements a nightmare
in the past.
2024-06-26 20:51:54 +02:00
Marvin Carstensen
8ec1695d1b Feature: support Tibber bridge as power meter interface 2024-06-26 20:50:50 +02:00
Bernhard Kirchen
83c59d7811 convert files with CRLF endings to LF endings
this only changes line endings. inspect this commit with command `git
show <commit-sha> --ignore-space-at-eol` and it will tell you that the
commit appears to be "empty" (since all changes are whitespace changes
near the end of a line, which are ignored in that git show command).

the files to be changed were found and updated using this command:
find lib src include webapp/src -type f | \
    xargs grep --binary-files=without-match --files-with-matches \
    $(printf '\r\n') | xargs dos2unix

the following files were restored afterwards, as they are using CRLF
line endings in the upstream as well:
 - lib/CMT2300a/cmt2300a_defs.h
 - lib/README
 - include/README
2024-06-24 21:57:12 +02:00
Andreas Böhm
79214d750f
refactor PylontechCanBattery to be more generic (#1064) 2024-06-23 19:42:04 +02:00
Bernhard Kirchen
034026f782 restore JK BMS dummy implementation
broke in be41e6b9. was unusable as the complete type of DummySerial has
to be available in the JK BMS controller header when defining the
unique_ptr managing the dummy instance. this problem is solved by moving
the whole dummy class into its own header.
2024-06-21 16:57:11 +02:00
Bernhard Kirchen
5e3a53d8d3 Merge upstream tag 'v24.6.10' into development 2024-06-21 16:45:41 +02:00
Andreas Böhm
83ac15405e
Feature: Implement DPL 'overscaling' to compensate shading (#956) 2024-06-20 13:32:29 +02:00
cerise21
ccbaf55808
Feature: Set/obtain DPL target power consumption via MQTT 2024-06-20 13:25:41 +02:00
cerise21
1f6fdb7fc0
Feature: Set/obtain DPL upper power limit via MQTT 2024-06-15 20:21:42 +02:00
Bernhard Kirchen
5a007e5352 fix: make SDM power meter use serial port manager
instead of hard-coding the use of hardware UART 2, the SDM power meter
instance now asks for a free hardware serial port to use and
instanciates the respective HardwareSerial object using said port.
2024-06-02 22:41:07 +02:00
Bernhard Kirchen
be41e6b906 refactor serial port manager: hand out UARTs FCFS
get rid of particular compile-time designations by UART index. just hand
out the next free index of hardware UARTs, or indicate that none is
available any more.

use names as keys to register and free UARTs.
2024-06-02 22:41:07 +02:00
Bernhard Kirchen
5a1c3af31f Feature: add support for a third Victron MPPT
only on ESP32-S3-USB. this fiddles with the available hardware UARTs to
make it possible to use a third Victron MPPT. if three MPPTs are defined
int the pin mapping, you will not be able to use the SmartShunt and JK
BMS battery interfaces.

note that using a second MPPT will also conflict with the SDM power
meter, and that conflict is not detected, yet.
2024-06-02 22:41:07 +02:00
Bernhard Kirchen
c22ae2bf8d Feature: show BMS FW and HW version (JK BMS, SmartShunt) 2024-06-01 11:23:22 +02:00
swingstate
561f4be6d6 Feature: SmartShunt: process midpoint voltage and deviation 2024-05-31 21:08:15 +02:00
Thomas Basler
6e607f7f67 Feature: Add option to clear eventlog at midnight 2024-05-31 00:07:28 +02:00
Thomas Basler
918c3449da Fix #2000: MQTT subscriptions where not updated if MQTT base was changed 2024-05-20 17:56:59 +02:00
Bernhard Kirchen
841523f028 embrace upstream __compiled_constants.h approach 2024-05-07 22:04:36 +02:00
Bernhard Kirchen
255e81e6bd Merge upstream tag 'v24.5.6' into development 2024-05-07 22:03:26 +02:00
Bernhard Kirchen
1dd64a57fd remove FirebaseJson lib from firmware (save 17.5k of flash)
we used this library solely to interpret the answer of an HTTP web
server as JSON and find a particular value using a path expression in
the HTTP power meter implementation.

since we ran out of flash memory on non-S3 ESP32, we need to cut some
corners. removing FirebaseJson is the last low-hanging fruit that we
currently know of. we can get rid of it by using ArduinoJson (which is
already integral part of the firmware) and implementing a custom logic
to extract a value based on a path expression.

other than the FirebaseJson path "finder", the new implementation
only knows how to access sub-keys delimited by a forward slash. in
particular, accessing array members is not supported any more. I am
hoping that this is simply not an issue. if so, we will have users
complaining and we can add this functionality in a later release.
2024-05-06 11:04:24 +02:00
eu-gh
686b5df64e
Feature: Publish Huawei AC charger mode via MQTT (#876) 2024-05-02 21:19:25 +02:00
Bernhard Kirchen
4cf596eb5a
Merge pull request #923 from helgeerbe/merge-v24.4.12
merge upstream tag v24.4.12, resolve conflicts (helgeerbe), fix eslint errors (schlimmchen) and adopt new web api method to save code duplication (schlimmchen).
2024-04-29 21:17:14 +02:00
Bernhard Kirchen
4e36c8c9ea
Feature: battery interface: use HW serial 0 on ESP32-C3 or S3 (#933)
this allows to use two VE.Direct interfaces, as there is no conflict
regarding HW serial port 2 after making the battery interfaces use
serial port 0 on devices with USB CDC. on those chips HW serial 0 is
free to be used since serial messages are written through the USB
interface directly.
2024-04-29 20:43:35 +02:00
helgeerbe
fdc5054480 Merge remote-tracking branch 'tbnobody/OpenDTU/master' into merge-v24.4.12 2024-04-25 20:59:19 +02:00
MalteSchm
f0df583c13
Feature: support for Huawei target power consumption 2024-04-24 20:26:56 +02:00
Bernhard Kirchen
74330a5617 Feature: restart unresponsive inverter
we found that the inverter sometimes stops responding to commands,
especially to the "start producing" command. we now count the number of
consecutive timeouts when trying to send a new limit or power state
commands. after two timeouts were recorded, every additional timeout
will send a restart command to the inverter.

as a last resort, if the counter keeps climbing, the DTU is restarted.

notice that this only targets unresponsive inverters which are
reachable. unreachable inverters are not restarted and do not cause a
DTU reboot. this is important for solar-driven inverters, which are
unreachable during the night. the DPL will not calculate a new limit and
hence the updateInverter() method will do nothing while the target
inverter is unreachable.

publish the timeout counter to MQTT for monitoring purposes.
2024-04-24 16:15:01 +02:00
Bernhard Kirchen
abe01ae36f Feature: HTTP power meter: support changing sign 2024-04-18 12:10:29 +02:00
Bernhard Kirchen
ede1abb5e6 Feature: HTTP power meter: support mW/kW as units 2024-04-18 12:10:29 +02:00
Bernhard Kirchen
e92701ccdf reuse power meter's HTTP config struct 2024-04-18 12:10:29 +02:00
Stefan Oberhumer
97800434c4 Prevent compiling the whole project on each commit.
Putting the git information into a generated sourcefile prevents
recompiling the whole project after each commit.
2024-04-18 09:02:57 +02:00
Bernhard Kirchen
4bc4defe66 DPL: insist on power meter value more recent than inverter stats
avoid performing a calculation based on a (slightly) outdated power
meter reading, which was aquired just before the limit was actually
applied by the inverter, but which was received by OpenDTU-OnBattery
after the inverter stats.
2024-04-15 23:02:58 +02:00
Bernhard Kirchen
52d7ac9581 Feature: DPL: support setups without power meter
without a power meter configured, the DPL now sets the base load as the
inverter limit if the battery charge allows it. it also takes
solar-passthrough into account, i.e., if the battery is in a charge
cycle but the solar output (Victron MPPT) is significant, the solar
power will be used up until the base load. if the battery reaches the
full solar passthrough threshold, the DPL will match the inverter limit
to the MPPT solar output.
2024-04-15 23:02:58 +02:00
Bernhard Kirchen
7e307114e5 Feature: DPL: introduce base load setting
on power meter issues (usually a timeout), keep the inverter enabled and
make it produce the configured base load limit if the battery can be
discharged. that should be okay since the base load config value is
expected to be small and a little less than the actual household base
load, i.e., if this amount of power is produced, the household will
consume it in any case and no energy is fed into the grid.
2024-04-15 23:02:58 +02:00
Bernhard Kirchen
cf1ea42f8b power limiter: remove obsolete enum 2024-04-15 23:02:58 +02:00
eu-gh
0ed09aeb4c Feature: Huawei: add SoC stop threshold and verbose logging switch 2024-04-07 17:16:50 +02:00
Thomas Basler
ea28903761 Move parsing of serial from web request to separate method 2024-04-05 19:14:56 +02:00
Thomas Basler
980e847ccb Feature: Check for out of memory situations when sending json responses
Also shows a nice message in the frontend if an internal error occours
2024-04-05 19:14:56 +02:00
Thomas Basler
2e3125fe8d Feature: Migrated ArduinoJson 6 to 7 2024-04-05 19:14:56 +02:00
helgeerbe
ff44267e73 Merge remote-tracking branch 'tbnobody/OpenDTU/master' into merge-v24.3.31 2024-04-03 11:53:03 +02:00
Bernhard Kirchen
92a7f27919 VE.Direct: use float rather than double
double precision floating point numbers are not needed to handle
VE.Direct values. handling double is implemented in software and hence
*much* more resource intensive.
2024-04-02 21:05:59 +02:00
Bernhard Kirchen
b299b9dc6c VE.Direct: simplify access to data
hand out const& to the data structs. this is possible now that this
struct is "stable", i.e., not reset regularly.
2024-04-02 21:05:59 +02:00
Bernhard Kirchen
ad125ea804 Fix: properly handle fragmented VE.Direct messages
queue every text event until the frame was checked by it checksum. then
process the data directly into the buffer struct. do not clear the
buffer struct, so it will always include the most recent value of a
particular data point.
2024-04-02 21:05:59 +02:00
MalteSchm
8abf614047 Feature: BMS initiated emergency charging
This change logically connects the AC-Charger with the BMS to add BMS
initiated emergency charging and respecting BMS current limits.
2024-04-02 21:05:00 +02:00
Thomas Basler
8add226a7c Save flash: Move WebApi json parsing to separate method to prevent a lot of redundant code 2024-04-01 13:52:09 +02:00
Thomas Basler
6f3b8fb8e1 Fix: Change default NTP server
Fixes #1877
2024-03-31 12:27:27 +02:00
Bernhard Kirchen
b8d0998a49 HTTP power meter: refactor tryGetFloatValueForPhase
unindent code, prepare this method for re-use on the same HTTP response
but with a different JSON path.
2024-03-23 22:13:59 +01:00
Bernhard Kirchen
054a677575 DPL settings in web app: split metadata from config
users are manipulating the DPL using HTTP POST requests. often they are
requesting the current settings using HTTP GET on the respective route,
then change a particular settings, and send all the data back using HTTP
POST. if they failed to remove the metadata node from the JSON,
OpenDTU-OnBattery would not be able to process the JSON due to its size.
the web app does not submit the metadata.

to avoid problems, the metadata is now split from the configuration
data.
2024-03-23 22:13:50 +01:00
Bernhard Kirchen
8bfb5c6523 implement and use Utils::checkJsonOverflow()
this method calls the overflowed() method on the respective
DynamicJsonDocument and prints a respective message if not all
data could be added to the DynamicJsonDocument.
2024-03-23 17:57:13 +01:00
Bernhard Kirchen
1c51c2de40 fix and harden BatteryStats update timestamp handling
* updating the SoC or value shall also update the general timestamp, as
  the latter is defined as "any value changed", which includes SoC and
  voltage, of course.
* if the last update is not a valid timestamp at all, the
  updateAvailable method must always return false, obviously.
2024-03-23 17:56:46 +01:00
helgeerbe
03435f66ee Feature: Set max number of supported Inverters to 10 (upstream default)
Due to the latest live view api changes, I revert the maximum number of supported inverters to the upstream default (10).
2024-03-22 13:50:16 +01:00
Bernhard Kirchen
b449dd1196 replace VICTRON_MAX_COUNT
determine the amount of controllers actually in use dynamically,
especially to avoid indices which are invalid, causing an error
to be printed, even though the user did not do anything wrong.
2024-03-20 22:11:11 +01:00
Bernhard Kirchen
8e5e8d169d fix: VE.Direct live data after supporting second MPPT
an error was made when determining whether or not to push a VE.Direct
MPPT's state through the websocket based on its data's age.
2024-03-20 22:11:10 +01:00
helgeerbe
5259fc172a Merge remote-tracking branch 'tbnobody/OpenDTU/master' into v24.3.15 2024-03-20 09:42:35 +01:00
Bernhard Kirchen
cd339a3a14 Feature: implement PowerMeter pin config for serial interfaces
in your pin_mapping.json, add a powermeter object like this:

[
    {
        "name": "My Board",
        ...
        "powermeter": {
            "rx": <num>,
            "tx": <num>,
            "dere": <num>
        },
        ...
    }
]

the SML power meter requires the rx pin to be set. the SDM power meter
requires the rx and tx pins are set. the "dere" pin pin is optional and
if set, this pin controls the driver enable and receiver enable pins of
the RS485 transceiver. the SDM library handles this pin.

closes #771.
2024-03-19 21:27:25 +01:00
PhilJaro
4f0385285c add new Victron SmartShunt values to liveView and MQTT (HASS) 2024-03-17 21:00:32 +01:00
Bernhard Kirchen
45c7243937 polish SMA HomeManager integration
* remove/comment unused variables to avoid compiler warnings

* cleanups: fix indention and style, make variable private, implement
  getters in header and make const.

* optimize message output: respect verbose logging setting. prefix
  output with SMA_HM.

* use newly introduced mutex in PowerMeterClass also for SMA HomeManager

* refactor code for readibility, unindent where possible.
2024-03-17 20:20:29 +01:00
Snoopy-HSS
f6680bd664 Feature: Support SMA HomeManager 2.0 as PowerMeter 2024-03-17 20:20:29 +01:00
Bernhard Kirchen
900326742c fix: prevent getTotalPower() reading intermediate results
the SDM power meter (among others) writes the power consumption of three
phases in multiple steps. this change helps to prevent getTotalPower()
reading intermediate values, e.g., reading a new value for phase 1 but
old values for phase 2 and 3 since phase 2 is currently read.

cache the values, and write them all at once, protected by a mutex,
later.

closes #732.
2024-03-17 16:54:48 +01:00
Bernhard Kirchen
13bc943dd5 Feature: Refactor/Simplify DPL settings
this changeset refactors the web application's DPL settings view. the
DPL settings can be complex, and they shall be presented in a way that
allows users to comprehend their meaning. irrelevant settings are now
hidden or displayed dynamically based on the influencing settings.

* group SoC thresholds into their own card

* hide battery SoC thresholds if battery disabled. if the user did not
  even enable the battery interface, battery SoC values will not be used
  for DPL decisions. in that case we completely hide the respective
  settings from the DPL admin view. this reduces the amount of settings
  for new users and especially users who don't even have a battery in
  their setup or have no BMS connected.

* group voltage thresholds and improve label texts

* fix load correction factor unit

* fix header (wording)

* group solar-passthrough settings in new card

* group inverter-related settings

* hide solar passthrough settings if VE.Direct is disabled. closes #662.

* completely disable form if any requirement is not met

* list available inverters by name and type. this makes it much more
  convenient to select the right inverter, especially since the order of
  the inverters in the web UI is decoupled from their position in the
  internal array, which was used to select them previously. care was
  taken that old configs select the same inverter after an update.
  when editing the DPL settings, the selects an inverter from the newly
  created drow-down list, and the respective old inverter is
  pre-selected.

* disable form if no inverter is configured (config alert)

* make inverter input selection dynamic. adjust selection to actual
  amount of channels for selected inverter. skip selection altogether if
  inverter has only one channel, or if it is solar powered.

* web app: wording adjustments

* group meta data into new property and exclude from submission. saves
  memory when evaluating the submitted settings.

* hide irrelevant settings if inverter is solar-powered

* move restart hour setting to inverter card. translate setting which
  disabled automatic restart.

* simplify "drain strategy" setting into an on/off toggle. care was
  taken that existing configs work the same after an upgrade. the
  respective drain strategy is translated into the new setting when
  reading the config. once the config is written, the new setting is
  persisted and the old is not part of the config any more.

* show more configuration hints, depending on actual configuration

* replace inputs by InputElement components where possible
2024-03-17 16:50:57 +01:00
Bernhard Kirchen
7d6b7252bf polish support for second VE.Direct MPPT charge controller
* fix compiler warning in SerialPortManager.cpp: function must not
  return void

* clean up and simplify implementation of usesHwPort2()
  * make const
  * overrides are final
  * default implementation returns false
  * implement in header, as the implementation is very simple

* rename PortManager to SerialPortManager. as "PortManager" is too
  generic, the static instance of the serial port manager is renamed to
  "SerialPortManager". the class is therefore renamed to
  SerialPortManagerClass, which is in line with other (static) classes
  withing OpenDTU(-OnBattery).

* implement separate data ages for MPPT charge controllers

* make sure MPPT data and live data time out

* do not use invalid data of MPPT controlers for calculations

* add :key binding to v-for iterating over MPPT instances
2024-03-17 16:50:15 +01:00
Arman Vartan
75541be248 Feature: Support for second Victron MPPT charge controller
this change adds support for a second Victron MPPT charge controller
using a second serial connection.

* Add device configuration for a second victron mppt
* Update VedirectView for second victron mppt
* Update MqttHandleVedirect for second victron mppt
* Update MqttHandleVedirectHass for second victron mppt
* Handle nonexisting victron controllers with optionals
* Add bool-function to Battery and inherited classes, if uart port 2 is
  being used
* Introduced a serial port manager. In order to prevent the battery and
  the Victron MPPT to use the same hw serial ports, this class keeps
  track of the used ports and their owners.
2024-03-17 16:50:15 +01:00
Bernhard Kirchen
7ebd4f4632 clean up defaults.h
* remove duplicated #defines. this is most probably a merge error from
  2024-01-16, as evidenced by 63205f88b, which added these duplicates.

* sort values by upstream and downstream projects. add a comment which
  tells us in the future where OpenDTU-OnBattery-specific values start.
2024-03-16 14:31:07 +01:00
Thomas Basler
33bf2117c6 Fix: Set all settings to default when deleting an inverters
Previously some old settings from previous inverters could have been shown in a new inverter
2024-03-13 18:47:28 +01:00
Bernhard Kirchen
56353e4f00 implement Battery::needsCharging()
currently this is only supported by the Pylontech battery provider, as
it reports a "charge battery immediately" alarm. this will also be
implemented by the JK BMS provider, and possibly also by the smart shunt
provider.

the method will be used to determine whether or not to start charging
the battery using the (Huawei) charger.
2024-03-13 17:07:05 +01:00
LukasK13
fba5c02346 Feature: Set powerlimiter thresholds via MQTT
publish DPL thresholds to MQTT, add support for setting powerlimiter
thresholds via MQTT, and make these auto-discoverable for Home
Assistent.
2024-03-10 22:10:02 +01:00
Bernhard Kirchen
91f8f61e63 Feature: DPL: explicit support for solar powered inverters
by default and until this change, we assumed that the inverter
controlled by the DPL is powered by a battery. not all users have a
battery in their system. they still use the DPL to achieve net-zero
export. those users can now tell the DPL that their inverter is powered
by solar modules rather than a battery and the DPL will behave
accordingly.
2024-03-09 15:40:13 +01:00
Bernhard Kirchen
b11b1dbcba DPL: define IsInverterSolarPowered config switch 2024-03-08 22:53:34 +01:00
Bernhard Kirchen
50635ee2ce Feature: live view: update with respective frequency
the update frequency of Victron MPPT charger data, the battery Soc, the
huawei charger power, and the power meter differ from one another, and
differ in particular from the inverter update frequency.

the OnBattery-specific data is now handled in a new method, outside the
upstream code, which merely call the new function(s). the new function
will update the websocket independently from inverter updates. also, it
adds the respective data if it actually changed since it was last
updated through the websocket.

for the webapp to be able to recover in case of errors, all values are
also written to the websocket with a fixed interval of 10 seconds.
2024-03-05 11:31:44 +01:00
Bernhard Kirchen
e432f0eca3 make BateryStats::updateAvailable wrap-around-safe 2024-03-05 11:31:44 +01:00
Bernhard Kirchen
8b6e57cda7 Fix: DPL: ensure inverter reaches requested state
we previously only called commitPowerLimit() if the desired limit
changed such that the change was bigger than the hysteresis. we found
that if the limit update was not received and the desired limit would
not change much, the limit of the inverter was wrong for a long time.

to mitigate this, we introduced re-sending the limit update every 60
seconds, regardless of what the limit reported by the inverter was at
that time.

if the power-up command was not received, we also would repeat it only
once every 60 seconds.

this leads to a new kind of staleness and the actual inverter state was
still not matching the desired state.

this new approach effectively adds an additional control loop at the
start of the DPL loop(). that new function compares the requested
inverter state to the actual reported state. it sends updates (limit
update or power on state) until the desired inverter state is reached,
or until a (hard-coded) timeout occurs.

this approach also allows us to send power-up, power-down, and limit
update commands independent from one another and in a particular order.

this should make sure that the inverter is in the desired state even if
conditions change slowly and commands were not received as expected.
2024-03-02 23:39:43 +01:00
helgeerbe
c72ae561c7 Merge remote-tracking branch 'tbnobody/OpenDTU/master' into development 2024-02-19 16:03:56 +01:00
Bernhard Kirchen
c930018764 Feature: DPL: use best available voltage value
the DPL is interested in the battery's voltage to make decisions about
draining the battery or letting it charge (if the user opts to use
voltage thresholds rather than SoC thresholds). using the DC input
voltage reported by the inverter under control has disadvantages:

* the data might be quite old due to the communication protocol
  implementation. more inverters being polled means even more lag. the
  connection being wireless makes this even worse, due to the need
  to retry the occasional lost packet, etc.
* the data is not very accurate, since the DC input of the inverter is
  actually some cabling and a couple of junctions away from the actual
  battery. this voltage drop can mostly only be estimated and is worse
  with higher load. the load correction factor is there to mitigate
  this, but it has its own problems and is cumbersome to calibrate.

instead, this change aims to use more accurate battery voltage readings,
if possible. the DPL now prefers the voltage as reported by the BMS,
since it is for sure the closest to the battery of all measuring points
and measures its voltage accurately regardless of the load (the voltage
reading will still drop with higher loads, but this will be only due to
the battery's internal resistance, not that of cabling or junctions). if
no BMS voltage reading is available, the DPL will instead use the charge
controller's voltage reading, as it is available with much higher
frequency and is assumed to be more accurate as it offers a resolution
of 10mV. only if none of these two sources can be used, the inverter DC
input voltage is assumed as the battery voltage.

closes #655.
2024-02-18 22:17:15 +01:00
Bernhard Kirchen
6df358242c Feature: know and use SoC precision
the Victron SmartShunt communicates the SoC value in permille. this
should be displayed in the web UI accordingly. this is a good excuse to
fully move ownership of the SoC value to the BatteryStats base class and
add a precision indicator variable. this is required to be set each time
a derived class (a battery provider) wants to update the SoC value. the
precision is then used when populating the JSON data for the web UI
(live view).

related to #573.
2024-02-18 22:17:15 +01:00
Bernhard Kirchen
7c069b1cc4 replace BatteryStats::isValid() method
in the respective context, the DPL only needs to be sure that the SoC
value is not outdated. it should not even care about other values
reported by the battery interface. hence, the isValid() method shall be
concerned with the SoC value timestamp only. the method is renamed for
clarity.
2024-02-18 22:17:15 +01:00
Bernhard Kirchen
3595725f8a Feature: implement subscription to battery voltage MQTT topic
this extends the MqttBattery implementation by an additional topic which
allows to subscribe to receive battery voltage readings through the MQTT
broker. similar to the battery SoC topic, this allows to import a
critical battery data point for the DPL, in case the user chooses to use
voltage thresholds rather than SoC thresholds to control the DPL. if an
otherwise incompatible BMS is available which publishes the battery pack
voltage through MQTT, this can now be used to feed accurate voltage
readings to the DPL.
2024-02-18 22:17:15 +01:00
Bernhard Kirchen
30bfffb848 BatteryStats: manage battery pack voltage in base class
the BatteryStats base class shall be able to tell the total battery pack
voltage. for that reason, and to avoid code duplication, the voltage is
now handled in the base class and treated as a datum that is common to
all battery providers.
2024-02-18 21:49:12 +01:00
Bernhard Kirchen
921302bf73 Feature: DPL: add switch allowing to ignore SoC
unfortunately, the battery SoC values reported by battery BMSs are
unreliable, at least for some users, or at least without regular
(manual) full charge cycles to calibrate the BMS. it offers great
advantages to connect OpenDTU-OnBattery to a BMS (MQTT publishing of
values, Home Assistent integration, etc.), but previously the users
were then forced to configure the DPL by SoC values.

this change allows to configure the DPL such that SoC values are
ignored. instead, the voltage limits are used to make DPL decisions, as
if no SoC was available in the first place.

the SoC related setting are hidden from the DPL settings view if SoC
values are configured to be ignored.

closes #654.
2024-02-18 21:46:42 +01:00
helgeerbe
b794f46ef0 Merge remote-tracking branch 'tbnobody/OpenDTU/master' into development 2024-02-09 20:14:58 +01:00
Bernhard Kirchen
1865113842 Feature: JK BMS Home Assistent integration
* pylontech HA integration: remove unused method/variable

* make MqttHandlePylontechHassClass::publishConfig() private.
  there are no outside users of that method.

* rename to MqttHandleBatteryHass

* battery HA integration: merge methods and bring back forceUpdate().
  even though the forceUpdate() method was not in use before, it makes
  sense to implement it and use it when the battery config changes.
  rather than controlling a separate flag, it now changes the _doPublish
  flag of the class, which also triggers publishing the device config to
  Home Assistant when an MQTT connection problem was detected. since
  both situations are now handled similarly, we can merge the loop() and
  publishConfig() methods.

* battery: provider specific sensors for HA

* move Battery MQTT loop to BatteryStats
  the BatteryStats class should handle the MQTT publishing, including the
  interval. for the calculation of a reasonable Home Assistent expiration
  value this class now also knows the maximum publish interval.

* JK BMS: fix publishing values for Home Assistent
  Home Assistent values expire, because we set them to expire after three
  MQTT publish durations. for that reason, we need to re-publish all
  values after our self-inflicted full publish interval.

* define JK BMS sensors for Home Assistent

closes #482.
2024-02-05 21:22:07 +01:00
Thomas Basler
c27ecc3620 BREAKING CHANGE: Web API Endpoint /api/livedata/status
To reduce the heap usage it is necessary to send the inverters one by one instead of a huge response. A simple call to `/api/livedata/status` returns just some very general information. If detailed inverter information are required the inverter serial number has to appended `?inv=<serial number>`.
The websocket also returns only one inverter at a time. It as to be assembled at client side.
2024-01-30 22:29:25 +01:00
Thomas Basler
557c5d645e Remove all files but the pin_mapping.json from filesystem on factory reset
This allows to create more user defined files and get them deleted on factory reset.
2024-01-30 22:29:12 +01:00
Thomas Basler
48a722f826 Merge branch 'pr1642' into dev 2024-01-30 20:47:45 +01:00
Stefan Oberhumer
e752c433af Use http header ETag caching for all static content.
Using the md5sum as ETag http header value should enable caching on all static http content.
2024-01-30 00:35:23 +01:00
Thomas Basler
2716f4c5df Removed not required pointer to AsyncServer instance 2024-01-27 01:09:55 +01:00
Thomas Basler
4e669d8932 Feature: Add support for ST7567 GM12864I-59N Display 2024-01-26 16:05:56 +01:00
helgeerbe
ebacc2f25f Merge remote-tracking branch 'tbnobody/OpenDTU/master' into development 2024-01-25 23:36:32 +01:00
Thomas Basler
4f2fbaaf2a Remove unused function parameter 2024-01-20 23:12:59 +01:00
Thomas Basler
7bc1a17fac Move task initialization from init method to constructor
This saves flash
2024-01-20 11:24:57 +01:00
Thomas Basler
251d197fb6 Migrate WebApi loop() methods to scheduler tasks 2024-01-20 02:00:22 +01:00
helgeerbe
f305b40be0 Merge pull request #612 from Fribur:development
- fix to HttpPowermeter not using explicitly specified non standard ports
- Revert back to using FirebaseJson instead of ArduinoJson
2024-01-17 13:26:52 +01:00
helgeerbe
63205f88be Merge remote-tracking branch 'tbnobody/OpenDTU/master' into development 2024-01-16 17:39:45 +01:00
Thomas Basler
c20caf8097 Feature: Support HMS/HMT inverters in different countries with different frequency bands
Thanks to @Fribur, @homeautomation2022 and @stefan123t
2024-01-14 16:37:34 +01:00
Thomas Basler
ee78698e37 Migrate all frequency calculations to Hz
Previously the code contains calculations  using a mixture  of kHz and Hz.

Thanks to @Fribur
2024-01-13 12:51:20 +01:00
Thomas Basler
f013698471 Feature: Added fullscreen diagram to display 2024-01-13 11:31:12 +01:00
Fribur
e136e096f4
Merge branch 'development' into development 2024-01-08 11:00:59 -05:00
Thomas Basler
637d4f06a3 Feature: Added option to disable the diagram at the display 2024-01-08 14:19:26 +01:00
Bernhard Kirchen
a012d81427
avoid too frequent SmartShunt data copies (#596)
currently the whole SmartShunt data structure is copied to the
BatteryStats instance in every loop, even though the data cannot
possibly have changed. this is quite an expensive task to do in every
loop. this change tracks the last update timestamp and only does the
copy operation if an actual updated data structure was received from
the smart shunt.
2024-01-07 18:30:02 +01:00
helgeerbe
ec93004724 Better handling of out of memory situations in live data websocket (onBattery) 2024-01-06 22:42:34 +01:00
helgeerbe
210fce67ce Use auto keyword and references more often (onBattery) 2024-01-06 22:10:07 +01:00
Thomas Basler
6573c51052 Use auto keyword and references more often 2024-01-05 22:04:54 +01:00
Thomas Basler
f00cd1bd61 Prevent empty HASS auto discovery topics if memory allocation fails 2024-01-05 21:55:15 +01:00
Thomas Basler
88d75673fc Prevent config corruption by checking whether memory allocation was successfull. 2024-01-05 21:55:15 +01:00
Thomas Basler
a65f1e48a5 Fix: Access Point not working after firmware update
Fixes #1613
2024-01-05 21:55:15 +01:00
Thomas Basler
2a15677923 Fix: Optimize network connection handling
This should provide a more reliable connection to several AP types. See #576
2024-01-05 21:55:15 +01:00
Fribur
e09ffcbb53 shorter parameter names 2024-01-05 14:39:32 -05:00
Fribur
85d0f2a8fb HttpPowerMeterClass: change order of resolving hostname
OpenDTU console gets spammed with "WifiGeneric::hostByName() error when first trying to resolve the hostname via DNS. So reverse order: first try mDNS, if that fails try DNS. Also ensure that https bool is passed correctly to HTTPClient::begin(). Lastly, concatenate strings for building Digest authorization using "+" and not via snprintf.
2024-01-05 14:36:19 -05:00
Bernhard Kirchen
3c8b8d4427
use frozen::string and frozen::map where reasonable (#593)
this change utilizes some of the features from library "frozen", which
was included upstream for the grid profile parser. to improve code
maintainability, a couple of std::maps mapping strings to values or the
other way around were introduced in OpenDTU-OnBattery-specific code at
the expense of some flash and computing overhead.

library "frozen" offers constexpr versions of map and string, which
saves initialization code and offers slightly faster lookups. this
brings the binary size down by ~25kB and should provide a small
performance improvement at runtime.
2024-01-04 23:28:34 +01:00
Fribur
f5c69060f5 re-factoring of HttpPowerMeter
Added ability to deal with local host names (mDNS), remove use of FirebasedJson to save ~20kB build size, some changes to PowerLimiter to avoid setting new inverter power limits when not needed (=current limit as reported by inverter is within hysteresis)
2024-01-04 16:20:32 -05:00
Bernhard Kirchen
e7a005839b
Feature: implement MQTT-driven battery provider (#589)
this battery provider implementation subscribes to a user-configurable
MQTT topic to retrieve the battery SoC value. the value is not
re-published under a different topic. there is no card created in the
web app's live view, since the SoC is already part of the totals at the
top of the live view. that is the only info this battery provider
implements.

closes #293.
relates to #581.
2024-01-04 15:42:10 +01:00
Thomas Basler
3c37b61f44 Use auto keyword and references more often 2024-01-04 15:38:56 +01:00
Thomas Basler
ca18d2c841 Prevent empty HASS auto discovery topics if memory allocation fails 2024-01-04 14:09:58 +01:00
Thomas Basler
4053e31a5e Prevent config corruption by checking whether memory allocation was successfull. 2024-01-04 13:42:39 +01:00
Thomas Basler
c1fc907ecb Fix: Access Point not working after firmware update
Fixes #1613
2024-01-03 22:04:50 +01:00
Thomas Basler
cbbe053cd5 Fix: Optimize network connection handling
This should provide a more reliable connection to several AP types. See #576
2024-01-03 01:17:39 +01:00
MalteSchm
5bd3ce5a8f
Reducing lower limit for AC-charger (#574) 2024-01-01 17:06:58 +01:00
helgeerbe
cbf7680836 merge v23.12.31 2024-01-01 14:50:51 +01:00
Bernhard Kirchen
8f5c4878c5
Fix: switch context when processing DPL MQTT requests (#572)
MQTT message callbacks are executed in the MQTT thread context. when
processing topics that control the DPL, we must avoid executing methods
that are not thread-safe. this change binds the methods to be called to
the respective parameters and executes them in the TaskScheduler
context, such that they no longer need to be thread-safe.
2023-12-31 14:49:39 +01:00
Bernhard Kirchen
463226082f clean up Huawei MQTT handler
* bind the callback to a topic (enum value) such that there is no need
  to tokenize the full topic (string) to find out what value is being
  processed. tokenizing is expensive.
* get rid of using the config in the callback, which improves
  thread-safety since the MQTT callback is running in the MQTT thread.
* prefer C++ method stof to convert MQTT value to a float, which saves
  us from using new and delete for a buffer in particular.
* prefer switch statements over if-else-trees.
* split long lines.
* get rid of topic #defines.
* fix indention.
2023-12-30 18:41:57 +01:00
Bernhard Kirchen
fe2f82e303 Fix: switch context when handling AC charger MQTT messages
MQTT message callbacks are executed in the MQTT thread context. when
processing topics that control the huawei AC charger, we must avoid
executing methods that are not thread-safe. this change bound the
methods to be called to the respective parameters and executes them
in the TaskScheduler context, such that they no longer need to be
thread-safe.
2023-12-30 18:08:06 +01:00
Bernhard Kirchen
7928f2f8cf
Feature: JK BMS: add more values to live view (#552)
there are more interesting values available to display in the live view.
however, adding them made the list of values very long. this can be
mitigated by using a new column/card, which uses the available screen
space nicely on bigger screens.
2023-12-30 16:46:44 +01:00
Bernhard Kirchen
c7098b6c42
Fix: thread-safety and dynamic memory for MessageOutput (#567)
this commit re-introduces the changes from #418, which were effectively
reverted with d49481097 (merge commit introducing TaskScheduler).

these adjustments are important to guarantee unmangled log messages and
more importantly, to guarantee that all messages from a particular
component are printed to the web console, which most people use to copy
messages from when reporting issues.

* use dynamic memory to allow handling of arbitrary message lenghts.
* keep a message buffer for every task so no task ever mangles the
  message of another task.
* every complete line is written to the serial console and moved to
  a line buffer for sending them through the websocket.
* the websocket is always fed complete lines.
* make sure to feed only as many lines as possible to the websocket
  handler, so that no lines are dropped.
* lock all MessageOutput state against concurrent access.
* respect HardwareSerial buffer size: the MessageOutput class buffers
  whole lines of output printed by any task in order to avoid mangling
  of text. that means we hand over full lines to the HardwareSerial
  instance, which might be too much in one call to write(buffer, size).
  we now check the return value of write(buffer, size) and call the
  function again with the part of the message that could not yet be
  written by HardwareSerial.
2023-12-30 16:45:56 +01:00
Thomas Basler
aa8068370e Feature: Add support for SSD1309 2.4" Display 2023-12-30 12:42:42 +01:00
helgeerbe
367e0f9b6e Merge remote-tracking branch 'tbnobody/OpenDTU/master' into development 2023-12-27 13:11:25 +01:00
helgeerbe
d494810975
merge V23.12.16 (#556)
* Optimize Sun data calculation

* Remove not required enum

* Split config struct into different sub structs

* Feature: Allow configuration of LWT QoS

* Made resetreason methods static

* Feature: Implement offset cache for "YieldDay"

Thanks to @broth-itk for the idea!
Fix: #1258 #1397

* Add Esp32-Stick-PoE-A

* remove broken LilyGO_T_ETH_POE config, use device profile instead

* Feature: High resolution Icon and PWA (Progressive Web App) functionality

Fix: #1289

* webapp: Update dependencies

* Initialize TaskScheduler

* Migrate SunPosition to TaskScheduler

* Migrate Datastore to TaskScheduler

* Migrate MqttHandleInverterTotal to TaskSchedule

* Migrate MqttHandleHass to TaskScheduler

* Migrate MqttHandleDtu to TaskScheduler

* Migrate MqttHandleInverter to TaskScheduler

* Migrate LedSingle to TaskScheduler

* Migrate NetworkSettings to TaskScheduler

* Migrate InverterSettings to TaskScheduler

* Migrate MessageOutput to TaskScheduler

* Migrate Display_Graphic to TaskScheduler

* Migrate WebApi to TaskScheduler

* Split InverterSettings into multiple tasks

* Calculate SunPosition only every 5 seconds

* Split LedSingle into multiple tasks

* Upgrade espMqttClient from 1.4.5 to 1.5.0

* Doc: Correct amount of MPP-Tracker

* Added HMT-1600-4T and HMT-1800-4T to DevInfoParser

Fix #1524

* Adjusted inverter names for HMS-1600/1800/2000-4T

* Add channel count to description of detected inverter type (DevInfoParser)

* Adjust device web api endpoint for dynamic led count

* Feature: Added ability to change the brightness of the LEDs

Based on the idea of @moritzlerch with several modifications like pwmTable and structure

* webapp: Update dependencies

* Update olikraus/U8g2 from 2.35.7 to 2.35.8

* Remove not required onWebsocketEvent

* Remove code nesting

* Introduce several const statements

* Remove not required AsyncEventSource

* Doc: Added byte specification to each command

* Feature: Added basic Grid Profile parser which shows the used profile and version

Other values are still outstanding.

* Optimize AlarmLogParser to save memory

* Add libfrozen to project to create constexpr maps

* Feature: First version of GridProfile Parser which shows all values contained in the profile.

* webapp: Update dependencies

* Apply better variable names

* Remove not required casts

* Add additional compiler flags to prevent errors

* Add const statement to several variables

* Replace NULL by nullptr

* Update bblanchon/ArduinoJson from 6.21.3 to 6.21.4

* Add const keyword to method parameters

* Add const keyword to methods

* Use references instead of pointers whenver possible

* Adjust member variable names in MqttSettings

* Adjust member variable names in NetworkSettings

* webapp: Update timezone database to latest version

* webapp: Beautify and unify form footers

* Feature: Allow setting of an inverter limit of 0% and 0W

Thanks to @madmartin in #1270

* Feature: Allow links in device profiles

These links will be shown on the hardware settings page.

* Doc: Added hint regarding HMS-xxxx-xT-NA inverters

* Feature: Added DeviceProfile for CASmo-DTU

Based on #1565

* Upgrade actions/upload-artifact from v3 to v4

* Upgrade actions/download-artifact from v3 to v4

* webapp: add app.js.gz

* Gridprofileparser: Added latest known values

Thanks to @stefan123t and @noone2k

* webapp: Fix lint errors

* Feature: Add DTU to Home Assistant Auto Discovery

This is based on PR 1365 from @CFenner with several fixes and optimizations

* Fix: Remove debug output as it floods the console

* Fix: Gridprofileparser: Add additional error handling if profile is unknown

* webapp: add app.js.gz

* Fix: Offset cache for "YieldDay" did not work correctly

* webapp: update dependencies

* webapp: add app.js.gz

* Fix: yarn.lock was outdated

* Fix: yarn build error

* Fix: Reset Yield day correction in combination with Zero Yield Day on Midnight lead to wrong values.

* Fix: Allow negative values in GridProfileParser

* Correct variable name

* Fix #1579: Static IP in Ethernet mode did not work correctly

* Feature: Added diagram to display

This is based on the idea of @Henrik-Ingenieur and was discussed in #1504

* webapp: update dependencies

* webapp: add app.js.gz

---------

Co-authored-by: Thomas Basler <thomas@familie-basler.net>
Co-authored-by: Pierre Kancir <pierre.kancir.emn@gmail.com>
2023-12-27 11:49:57 +01:00
Stefan Oberhumer
ac427523b5 Feature: Add screensavermode also to diagram drawing. 2023-12-24 12:24:10 +01:00
Thomas Basler
0ddc7fd28d Feature: Added diagram to display
This is based on the idea of @Henrik-Ingenieur and was discussed in #1504
2023-12-19 17:26:24 +01:00
Thomas Basler
84248ec9b6 Feature: Add DTU to Home Assistant Auto Discovery
This is based on PR 1365 from @CFenner with several fixes and optimizations
2023-12-17 14:40:04 +01:00
Bernhard Kirchen
6e78c5bd1c
Feature: JK BMS: export (more) data to live view and MQTT (#549)
* add more values to web app live view. this should add all interesting
  values for the web app live view. those include important values and
  values that change frequently.

* add more interesting JK BMS dummy messages: one has 0% SoC and an
  alarm (discharge undervoltage) set. the other has the undertemperature
  alarm set.

* add alarms and warnings to live view

* publish alarm and status bits through MQTT individually

* publish cell voltages to MQTT broker

* remove trailing spaces in BatteryStats class
2023-12-15 10:59:07 +01:00
Thomas Basler
b4ead9d6ec Feature: Allow setting of an inverter limit of 0% and 0W
Thanks to @madmartin in #1270
2023-12-13 22:15:49 +01:00
Thomas Basler
4f7597eaa1 Adjust member variable names in NetworkSettings 2023-12-12 23:55:03 +01:00
Thomas Basler
66530638de Adjust member variable names in MqttSettings 2023-12-12 23:54:58 +01:00
Thomas Basler
e95afbf2cc Use references instead of pointers whenver possible 2023-12-12 13:26:07 +01:00
Thomas Basler
8fb43fedf8 Add const keyword to methods 2023-12-12 01:30:40 +01:00
Thomas Basler
d6494fa791 Add const keyword to method parameters 2023-12-12 00:21:14 +01:00
Thomas Basler
5e45006831 Replace NULL by nullptr 2023-12-11 20:23:45 +01:00
Thomas Basler
06651f365a Feature: First version of GridProfile Parser which shows all values contained in the profile. 2023-12-11 14:58:17 +01:00
Thomas Basler
b937532505 Remove not required AsyncEventSource 2023-12-07 15:21:24 +01:00
Thomas Basler
8b5d406a4f Introduce several const statements 2023-12-07 15:19:04 +01:00